LEDE Table of Packages: Good or bad?

(spin off of Toward a good "Flashing LEDE instructions" page)
Currently LEDE does not allow easy access to what packages are available and what they do if you don't have a LEDE system on-hand to run opkg to list packages/descriptions.

So the most straightforward way to show newcomers what we have in LEDE is to tell people "install LEDE (on a device or in Virtualbox) and then see what packages there are", which is a bit backwards.
The only alternative is looking at the package names in the folders of the download server, or downloading the package lists, uncompress them and read a long and boring list. Not cool.

I was thinking about having a Table of Packages too in the wiki, something similar to the ToH but loaded with basic information about packages (automatically, by a script, I can make the script).
It might also link to package-specific articles if needed.

I'll be honest though: I usually spend my time on the wiki to add content in the user guide and documentation so that is probably yet more work for tmomas or someone else that likes the idea. I can make the script to update the system though.

:runner: (looks around and then runs away stealthily)

1 Like

I highly doubt that an automatic long list / database of all available packages would be useful for newcomers. There are thousands of packages/variants/modules, many of which have only short and non-descriptive titles & descriptions, while others have long "advertising-style" descriptions. So, the data that can be automatically collected, would be non-uniform. The availability of the packages varies somewhat by the architecture/platform, which also causes complexity to the possible database/script.

Some of the packages can be useful for most users (e.g. USB storage support, router statistics, QoS, ...), while some (e.g. programming tools for a specific rare microcontroller) are really niche tools and will be used by less than 0.01% of the user population. So, dumbly listing all packages with equal importance would not be really useful for newcomers.

I think that a categorized "Top picks" list for 20-50 interesting topics would offer more value to the beginners than a large searchable package database.

Page could be something like "Extend the device's functionalities with add-on packages". It could have maybe 20-50 categories (USB storage, statistics, web proxy, adblocking, VPN, QoS, P2P, USB modems, file server, home media server, multimedia etc.) and each category could have a link to a possible "How-to" recipe/topic page of that topic (if such page exists) and a list of 1-5 most widely used / most useful packages for that topic.

Something like an improved version of the (rather well hidden) Openwrt wiki how-to front page https://wiki.openwrt.org/doc/howto/start and the how-to index https://wiki.openwrt.org/doc/howto/index.by.subject
Note that the Openwrt how-to pages are also a bad mix of high-level topic pages & really niche router hardware config advice, so they are not perfect at all.

[quote="hnyman, post:2, topic:123, full:true"]I highly doubt that an automatic long list / database of all available packages would be useful for newcomers.[/quote]The ToH's database system main use is making many tables (called "views") where the content of the same large database is already filtered heavily to show only X or only Y.

The monstrously huge table is mostly for mantainers or people testing their massive multiscreen setup.

If you want to make a table in a wiki article showing only USB, basic routing, whatever, you write like 5 lines to summon the data plugin and tell it what you need there, and the plugin does the leg work in reading the database, fetching the right information according to your filters and showing it.

And this is kept updated automatically, you just update the central database, and every other table downstream will be up-to-date.

[quote]There are thousands of packages/variants/modules, many of which have only short and non-descriptive titles & descriptions, while others have long "advertising-style" descriptions. So, the data that can be automatically collected, would be non-uniform.[/quote]Having a way to show package descriptions like that will encourage better descriptions from the package mantainer, and normal people might start sending "improved description" PRs through Github's webinterface.

[quote]The availability of the packages varies somewhat by the architecture/platform, which also causes complexity to the possible database/script.[/quote]Architecture would just be another field in the database (shown or not depending on need, surely used for filters).
It might be necessary to read the makefiles in the source instead than the actually downloadable package lists to get that info faster, though. (I assume that if it works for the build system it will work for a script loading a database too)

[quote]Page could be something like "Extend the device's functionalities with add-on packages". It could have maybe 20-50 categories (USB storage, statistics, web proxy, adblocking, VPN, QoS, P2P, USB modems, file server, home media server, multimedia etc.) and each category could have a link to a possible "How-to" recipe/topic page of that topic (if such page exists) and a list of 1-5 most widely used / most useful packages for that topic. [/quote]Yes, I was thinking about making something like this, with different pages for different categories. Just that the tables in the articles would be auto-generated according to my filters.

Having the packages in a database would allow also to automate the "most used" selection. Just add a field in the database to mark the package as "most-used" and you can auto-build and maintain tables with only "most-used" packages, reducing the need for manual editing.

The same can be done to packages "marked-for-deletion", or "from-a-third-party-repo" or "LEDE-trunk" or "LEDE-first-release-branch" or "LEDE-second-release-branch" or for whatever reason.

The goal is to get something like most PC Linux distros have, like for example Debian, https://packages.debian.org/stable/
(Debian has a relatively simple system with large lists of packages, but you should note the basic features of a webpage showing packages)

The input for a Table of packages would be simple .txt files per package that reside in /pkgdata/

_pkg-category, _pkg-arch and _lede-release define all possible values for these dropdowns.

If you can provide the package dataentries according the below example, this can be set up in almost no time. Feel free to modify the example as needed, it is just a quick proposal.

Example: /pkgdata/airpwn.txt

---- dataentry packages ----
Package name_pkg-page      : airpwn # Name of the package
Package version            : 1.3-1 # Version of the package
Package description        : Airpwn is a framework for 802.11 (wireless) packet injection. Airpwn listens to incoming wireless packets, and if the data matches a pattern specified in the config files, custom content is injected "spoofed" from the wireless access point. From the perspective of the wireless client, airpwn becomes the server. # Description, max. 1kB
Package size               : # .ipk size in kB, without dependencies!
Dependencies               : # List dependencies of this package
Categories_pkg-category    : wireless, network, ... # Select multiple categories
Architectures_pkg-arch     : # Select multiple architectures
LEDE releases_lede-release : 16.12, 17.06, 17.12 # Select multiple releases
----

This posting is a "wiki": Feel free to edit this example as needed!

I also would like to see a WIKI section dedicated to packages, however I agree with hnyman that just reprinting the text in the packages file is not sufficient. I can just go to Software in Luci and get this.

I have been frustrated by the poor descriptions included with most packages, and the general lack of install, config and use documentation for all. Yes, some packages, like a file system, require little explanation. Others like miniDLNA have a bit of config and strategy to their setup. Still it would be nice to know developer, date info, etc.

[quote="RangerZ, post:6, topic:123, full:true"]I agree with hnyman that just reprinting the text in the packages file is not sufficient.[/quote]each package can/should link to a specific page, much like the ToH links to the device page.

[quote]I can just go to Software in Luci and get this.[/quote]Think of people that don't have installed LEDE (yet) or don't have a LEDE box on hand, they don't even have that.

Really, when the best local source of info about packages is keeping a clone of the git repo and doing a "make menuconfig" from terminal, something is very wrong.

[quote]I have been frustrated by the poor descriptions included with most packages,[/quote]The description is easy to change from Github's web interface (if you have a github account anyway)

You go here https://github.com/openwrt/packages/blob/master/utils/avrdude/Makefile

and you see the description in this block of the makefile of each package (sitting in a folder called like the package itself)

define Package/avrdude/description 
    AVRDUDE is a full featured program for programming Atmel's AVR CPU's.      
endef

You push the edit button (a pencil icon on the top right of the document), you edit it and then you write below what you did and why, title must have : at the start.

Then you click "Propose File Change" and then it should show another page where there is a button with "Create Pull Request". Will open up another message box with instructions you should follow to make a valid pull request message (like notifying the package mantainer).

Then you must wait for the package mantainer to show up and agree with the change, then a repo mantainer commits the change and it goes live. (or a repo mantainer commits the change regardless of the fact that the package mantainer showed up if he feels it is ok)

[quote]and the general lack of install, config and use documentation for all.[/quote]Heh, that is because none ever took the time to write down some tutorial about them, the ToP isn't going to change that by itself, but will sure put packages more in the spotlight.

[quote="tmomas, post:4, topic:123, full:true"]

The input for a Table of packages would be simple .txt files per package that reside in /pkgdata/

_pkg-category, _pkg-arch and _lede-release define all possible values for these dropdowns.

If you can provide the package dataentries according the below example, this can be set up in almost no time. Feel free to modify the example as needed, it is just a quick proposal.
[/quote]Sounds easy, I'll try it out later.

It seems to me that we should focus our efforts on helping authors promote package that are popular/useful/etc. (as opposed to attempting to generate a sensible catalog of all the packages that have ever been created.)

A good HOWTO page (and that is well-displayed) helps readers decide what packages are available.

The wiki provides an "automatic filter" for good packages: it'll display packages where package creators add them. (If the package creator can't summon up the energy to make a howto/documentation, then I don't feel strongly that we need to spend our time publicizing their minor package.)

[quote="richb-hanover, post:9, topic:123, full:true"]The wiki provides an "automatic filter" for good packages: it'll display packages where package creators add them. [/quote]You are probably oblivious to the most common reason someone adds packages to OpenWRT package feeds (or contributes to opensource projects in general).

The average package contributor adds a package to OpenWRT community feeds because HE or his downstream pet project needs it, and on average he knows how to use it already.

As history has shown with OpenWRT wiki, pretty much none of such package contributors takes the time to go and write down documentation for them (there is some stuff for VPN, some stuff for 3G connectivity, something about multiwan, some incoherent and outdated ramblings about something else, and that's pretty much it, in any case the only one that seems to have documentation written by the package contributori is StrongSwan as it has that info also on its website)

Meanwhile in the package feeds there are like 400 packages doing various stuff none knows about.

A good HOWTO page (and that is well-displayed) helps readers decide what packages are available.

  1. install LEDE
  2. opkg update (or update from within luci)
  3. opkg list | grep "keyword" ( or search from within luci)

Really, the bigger issue is the #1.

Why to know what packages I can install do I need to install LEDE first (or download source code)? On what OS or platform is that acceptable?

[quote](If the package creator can't summon up the energy to make a howto/documentation, then I don't feel strongly that we need to spend our time publicizing their minor package.) [/quote]Are you a friend of tmomas perhaps (he seems to behave the same towards OEMs that don't provide proper hardware specs)?
You aren't punishing the package contributors as they know how to use their package already. You are only hurting LEDE community of people that might want to know what stuff is in there and might contribute to the wiki better documentation for it.

More people know what is in -> more people want to use that stuff -> more chances someone will find the time to put down a half-decent tutorial and documentation for the package he is using.

Note that I'm talking about setting up infrastructure that facilitates this kind of community contributions, not saying we (wiki mantainers) should waste time to add tutorials ourselves.

[quote="tmomas, post:4, topic:123, full:true"]The input for a Table of packages would be simple .txt files per package that reside in /pkgdata/

_pkg-category, _pkg-arch and _lede-release define all possible values for these dropdowns.[/quote]Ok I found the /pkgdata and the example package file. Where are the other three you mention?

[quote]If you can provide the package dataentries according the below example, this can be set up in almost no time. Feel free to modify the example as needed, it is just a quick proposal.
[/quote]I'm unsure on how you made up the names in your example (you used spaces, "_" and "-", does that mean anything?). Or where should I add more of such entries.

Can you be more specific on the naming convention there?

Anyway, this is roughly the info my script will format in the way you need for each package (this is a single package entry of opkg package lists)

Package: 6in4
Version: 24-1
Depends: libc, kmod-sit, uclient-fetch
Source: feeds/base/package/network/ipv6/6in4
License: GPL-2.0
Section: net
Maintainer: Jo-Philipp Wich <jo@mein.io>
Architecture: all
Installed-Size: 1476
Filename: 6in4_24-1_all.ipk
Size: 2283
MD5Sum: 38edcde88ab92cef289f14857128a1fe
SHA256sum: 0c2fdd688ce078a874c6212414348da71c534f0fade113f71a4c8bed02695684
Description:  Provides support for 6in4 tunnels in /etc/config/network.  Refer to http://wiki.openwrt.org/doc/uci/network for configuration details.

Plus it will add some other field to categorize better the package (probably using the SUBMENU field from makefiles, the field used for categorizing the packages in the make menuconfig interface), a link to this package's github page (to encourage contributions), and a link to a LEDE wiki article about it.

Also, just noticed that the other post with the example is editable by everyone. Neat, we can make wiki posts too here?

EDIT: removed leftovers of initial post.

Oh yes, they mean something :slight_smile:We had long discussions regarding them in the current OpenWrt dataentries.
Detailed info: https://www.dokuwiki.org/plugin:data#data_entry_input

Example: Categories_pkg-category
Categories = Fieldname how it later also appears in the datatables under "cols"
_pkg-category = datatype of this field -> can be set in the admin area only; this datatype provides the possible values of the dropdown; same for _pkg-arch and lede-release
'-' in pkg-category = separator, instead of '
' which is already used to separate datafield from datatype

Dependencies vs. Dependenciess vs. Dependencies_
Dependencies : the last 's' means "multiple values possible"; shown as "Dependencie" in a datatable
Dependenciess: the last 's' means "multiple values possible"; shown as "Dependencies" in a datatable
Dependencies_: single value only; shown as "Dependencies" in a datatable

It's confusing, I know %-)

Befor you create the txt files, let me see an example, in order to get the naming right.

Is there any thing that can be done, going forward, that sets a standard/definition/process to the documentation that is minimum/required for LEDE to accept a new package? See number 5 here

[quote="RangerZ, post:13, topic:123, full:true"]what we want IN LEDE. Richb-hanover's point is good. Paretos law (80/20 rule). [/quote]we have no right to decide what should be shown and what should not. We are just contributors to a larger project, not the leaders (knowing a bit LEDE goals, I doubt they would limit that either).

Sure you can contribute only on the parts you feel like contributing, but making choices on behalf of LEDE on what is worthy and what is not of being shown at all... nope.

All packages go in the list, no exceptions. Then you can make wiki pages with "most used packages" and they will be certainly useful, but not showing some packages because they are "niche" is silly.

What I'm doing is still going to be much faster than adding even a list of like 50 packages by hand, and is much more maintainable.

[quote]Is there any thing that can be done, going forward, that sets a standard/definition/process to the documentation that is minimum/required for LEDE to accept a new package? See number 5 here[/quote]Keep in mind that most packages in LEDE come from OpenWRT community package feeds, so LEDE hasn't direct control over that.

But yeah, adding a "you must write some basic documentation for the wiki" rule for PRs would be cool and would make sense. Something is already enforced, but the descriptions of many PRs are very short, lack examples and so on (it's an explanation of what is done to help the review of the code more than true documentation).

I'd say to open a new thread about this and see what LEDE core people (the ones with commit access) say, as they will be the ones enforcing it.
Also quite a bit of OpenWRT community package feeds mantainers are also here on LEDE (hnyman and thess for example), so they might also do something for the package feeds too.

[quote="tmomas, post:12, topic:123, full:true"]Before you create the txt files, let me see an example, in order to get the naming right.[/quote]Here the example:
---- dataentry packages ---- Name_pkg-page : # Name of the package Version_ : # Version of the package Description : # Description, max. 1kB Installed size_ : # installed size in kbytes, without dependencies Dependenciess : # List dependencies of this package Categories_pkg-category : # Select multiple categories Architectures_pkg-arch : # Select multiple architectures LEDE releases_lede-release : # Select multiple releases File name_ : # File name of the package File size_ : # .ipk size in kbytes, without dependencies MD5Sum_ : # Md5sum if any. SHA256sum_ : # SHA256sum if any. License : # License type. Mantainer : # Mantainer name (also email?) Wiki link_ : # link to wiki page, filled automatically by the script ----

I tried to keep most fields as a single value.

What you think about showing the maintainer email? I think it might be better to hide it in the package lists in the wiki.

I assume that the spaces in the file aren't relevant, only thing that matters is that there is a "field name" : "field value".

---- dataentry packages ----
Name_pkg-page              : # Name of the package
Version                    : # Version of the package
Description                : # Description, max. 1kB
Package size               : # raw .ipk size in kbytes, without dependencies
Dependenciess              : # List dependencies of this package
Categories_pkg-category    : # Select multiple categories
Architectures_pkg-arch     : # Select multiple architectures
LEDE releases_lede-release : # Select multiple releases
File name                  : # File name of the package
File size                  : # .ipk size in kbytes, without dependencies
MD5Sum                     : # Md5sum if any.
SHA256sum                  : # SHA256sum if any.
License                    : # License type.
Maintainer_pkg-maintainer  : # Maintainer name
Maintainer_title           : contact|<maintainername> # Maintainer name
----
  • Wiki link_ is not needed, that function is available via Name_pkg-page. This will link to /packages/Name
  • Version doesn't need the '_', since there's no 's' at the end. Same for other fields without 's'at the end.
  • Installed size: AFAIK there's no way to predict the installed size, therefore I would take the raw .ipk size, since that's the only thing you really know.
  • Maintainer_pkg-maintainer: This will link to a personal page like https://wiki.lede-project.org/user/tmomas; prerequisite: maintainer name = the same as in the wiki
    Alternatively, this could link to "contacts" in the wiki; no email needed for this.
    Maintainer_title : contacts| # Maintainer name#

Sidenote: Strangely, the display of bulleted lists depends on whether there is some text following the list or not. In preview, the bullets are shown if there's no following text after the UL; in thread view, the bullets are only shown when there is some following text after the UL.

Spaces as shown above are optional. They are convenient if you are working with the wikisource of the pages, instead via the dataentry editor. This layout can easily be implemented via "here documents" in bash.

"field name" : "field value" # Comment
The comment is shown when you edit the dataentry via the editor. Regard it as a help or for the user editing the dataentry or a hint what is expected in this field.

[quote="tmomas, post:16, topic:123, full:true"]* Installed size: AFAIK there's no way to predict the installed size, therefore I would take the raw .ipk size, since that's the only thing you really know.[/quote]The installed size comes directly from the opkg package lists (see above, for each package it lists the "installed size" too). I assume that it is calculated in the package building process by applying the same compression algorithm used in jffs2 (or ubifs for that matter, it's the same afaik), as that number should be used by opkg to avoid trying to install packages when there is too little space left (or at least it was, I didn't check recently).
Squashfs compression is likely going to be better than that, yes, so maybe I need to change the name for that field.

[quote]* Maintainer_pkg-maintainer: This will link to a personal page like https://wiki.lede-project.org/user/tmomas; prerequisite: maintainer name = the same as in the wiki
Alternatively, this could link to "contacts" in the wiki; no email needed for this.
Maintainer_title : contacts| # Maintainer name#[/quote]I'm not including this, as I'm not looking to make a contact system, most package mantainers are from the OpenWRT community package feeds and don't have a Wiki username anyway.

OK, then make that "installed size" again.

Regarding "Maintainer_pkg-maintainer"

I would find it very convenient if there was some kind of maintainer contact / bugreport link available.
Take https://www.dokuwiki.org/plugin:data as an example. This provides a bug report link and an email link to the author. If someone is the maintainer of a package, he should be reachable somehow, or at least the user should be given a link where to report bugs to.

[quote="tmomas, post:18, topic:123, full:true"]If someone is the maintainer of a package, he should be reachable somehow, or at least the user should be given a link where to report bugs to.[/quote]Ok, will add a "report bug" link that goes to the LEDE bugtracker if the package is from LEDE core, or to OpenWRT community package repos issue page if the package is from there.

I have been following the OpenWrt community for ~18 months, and I would say your note (below) perfectly sums up the state of the OpenWrt wiki.

I want to back up a little bit and ask some overall questions before addressing the specifics here. I've put them in the Site Feedback category for your consideration.