Definitely useful. At least for package administration etc. stuff. E.g. @bobafetthotmail has been doing lately a lot of menuconfig reorganisation and this kind of overview helps finding anomalies.
You have to be careful, though.
The category indices are created via fulltext search. Unfortunately, I can't use regex in the pagequery-plugin search, but only simple search queries. If you look at the wikisource of the individual index pages, e.g. https://wiki.lede-project.org/packages/index/administration, you will notice that the search strings may also be found somwhere else in a dataentry, e.g. in the description, rather than only in the category line.
search strings: (": administration," | ", administration," | ", administration #")
Therefore there might be some additional packages here and there, compared to the numbers shown in the Package categories datacloud on the Packages page.
Not a 100% solution, but close to. 
A minor flaw in the dataentry pages: Same category twice in one dataentry, see e.g.
https://wiki.lede-project.org/packages/pkgdata/luci-i18n-mmc-over-gpio-el?do=edit
Categories_pkg-categorys : luci, luci
Interestingly, you see this only in the wikisource, not in the dataentry itself nor in the datatables.
These package descriptions are coming along well.
Is there any way to include a "Last Updated:" field in the descriptions? It would great to help people understand whether a package is up to date, or long-abandoned. Thanks.
Another thought: The package dataentry page layout is confusing. The legends ("Name:") are intermingled with the data. See for example: https://wiki.lede-project.org/packages/pkgdata/6in4
Their HTML structure is similar to the device techdata page (https://wiki.lede-project.org/toh/hwdata/netgear/netgear_d7800). Both show:
<dl>
<dt>...
<dd>...
Is there a way to make the formatting of Package pages more pleasing? Thanks.
I don't see any intermingling.
OK, I've seen it now and fixed it via CSS (at least down to 419px window width).
[quote="tmomas, post:96, topic:123, full:true"]Just noticed: Some empty strings in the description.[/quote]That's actually not my fault, the description is like that in package lists too. Only packages coming from one specific makefile in the source have this issue, most other packages with a similar setup (asterisk11 stuff) don't have this issue.
Will have to report this.
Can we live without the "File name"?
Yeah, dropped.
There are 2 spaces between the last category and the '#'
fixed, this wasn't the only place where there were double spaces, btw. All places where the name was read by parsing text needed an additional sed command to remove all trailing spaces.
n general: We don't need any comments (=everything after the #) due to the read-only status of the dataentries. But of course we can also leave them in, should someone feel the need to read the wikisource of the dataentries.
dropped most of the comments, left only those for less-clear entries.
I got the automatic indexing of dataentry pages working now:
That's great. It mimicks Debian's, I think something like that should be the default package view instead of the table, while the tables are used only in specific pages about some topics, or for the "search packages page".
Might be worth considering a similar setup also for devices.
Only complaint is that the text is a bit tiny.
Unfortunately, I can't use regex in the pagequery-plugin search, but only simple search queries. If you look at the wikisource of the individual index pages, e.g. https://wiki.lede-project.org/packages/index/administration, you will notice that the search strings may also be found somwhere else in a dataentry, e.g. in the description, rather than only in the category line.
I can add a dedicated field to the package txt where I use a special name, like "administration" -> "package_administration" so you can use a search string and will find all such files.
A minor flaw in the dataentry pages: Same category twice in one dataentry, see e.g.
That's the repo name (luci) and the category (luci again). I've split the repo name in its own field as it's not a real category we care about, but might be useful when searching or for secondary tables.
I also altered the categories like this (example)
network, network -> linux atm tools
So this package will be in both "network" and "network -> linux atm tools", as now the submenu name shows also the parent category.
If the "->" cannot be used, please suggest a symbol I can use instead.
[quote="hnyman, post:101, topic:123, full:true"]Definitely useful. At least for package administration etc. stuff. E.g. @bobafetthotmail has been doing lately a lot of menuconfig reorganisation and this kind of overview helps finding anomalies.[/quote]Actually, this project is a spin-off of that. I didn't finish there, but I saw I needed better tools to look at what I'm doing.
And of course some kind of stronger argument just in case, as someone might see reorganization of the make menuconfig as a minor thing/annoyance, but will be more interested if the same tags are also used to make tables in the wiki.
[many changes to the package dataentries]
Wonderful! Can I have the updated dataentry pages for implementation in the wiki, please?
As previously discussed, a single *.tgz or alike uploaded via the media manager would be perfect.
(hmmm... I think all this would be easier if you had ssh access...)
I can move the index https://wiki.lede-project.org/packages/index/start to https://wiki.lede-project.org/packages/start, and the datatable to some other pagename (e.g. /packages/top), linked to from the index (or anywhere else).
Regarding font size: enlarged from 90% -> 120%.
Please let me know if this is OK.
I'm not yet sure that we need additional fields. The current fulltext search is not that bad and mostly fits to the numbers shown in the datacloud. On the other hand, and underscore here and there would perhaps make the search more exact. See yourself what the indexpages return after the new dataentry import, and play around with the pagequery and/or category naming if something doesn't work as expected.
[quote="tmomas, post:110, topic:123, full:true"]Wonderful! Can I have the updated dataentry pages for implementation in the wiki, please?
As previously discussed, a single .tgz or alike uploaded via the media manager would be perfect.[/quote]I'm still implementing the part of the script that reads/adds base packages and kmod- packages (as that does need some custom things).
Will update wiki pages when I upload the files.
[quote](hmmm... I think all this would be easier if you had ssh access...)[/quote]I'm able to control linux systems through ssh (or plain terminal anyway). You might want to make a non-root account with access only to folders that matter for this usage though.
[quote]I can move the index https://wiki.lede-project.org/packages/index/start to https://wiki.lede-project.org/packages/start, and the datatable to some other pagename (e.g. /packages/top), linked to from the index (or anywhere else).[/quote]Yeah, something like that. Don't do it now though as I will likely need at least another week for the package reading script to get to the final state, hoping jow makes a release soon so I can also test the part for reading/indexing multiple LEDE releases automatically (so that when there is a new release you don't need to go and add things manually in the script).
[quote]Regarding font size: enlarged from 90% -> 120%.
Please let me know if this is OK.[/quote]Ah nice it's possible to increase size.
I'd increase it a bit more, like the current size of "package categories" title in that page. (140% maybe?).
@thess Can you arrange ssh access for Alberto? To create and edit files in the dokuwiki, he would also need sudo rights. He would be a great help in implementing an automatically updated and user-friendly display of available packages in the wiki.
With CSS, you can do everything but cooking pudding.
To get it right:
Current status:
https://wiki.lede-project.org/packages/index/start
https://wiki.lede-project.org/packages/index/network
-> both, category index and network index, have the same font size (120%)
You want:
a) both to be 140%
b) the category index to be 140%, but for the more detailed package index (in this eample network) only the current 120%
I find b) more appealing in terms of quick overview. Less fontsize -> more packages per display area, especially for big categories like network or community packages with 1200 / 2200 packages.
Quick update:
AAAAAAARGH. 
LEDE packages with kernel modules (drivers) have 3 different versions and different kernel version dependencies as in LEDE we have 3 different kernel versions (currently 3.18, 4.1 and 4.4).
And also architectures are different (as some archs/devices are on 3.18, some on 4.1 and most on 4.4).
Also architectures are different again as these packages come from a special repo with the same name of their target/subtarget device, not common package repos used by many different devices.
Different kernel versions usually mean different level of features supported by kernel/drivers, so I would like to show this.
I'm really scratching my head on how to get this indexed in a sane way without adding dedicated fields and requiring babysitting whenever there are version changes (like say if someone adds devices at kernel version 4.7 and we then have 4 different versions instead of 3 then someone would have to change all wiki tables to add new fields to show it)
Ideas welcome. So far my only idea is adding the (short form of) kernel version to the text file name so I basically triplicate kmod packages (they are 580 or so, not all would be triplicated, but most would).
I find b) more appealing in terms of quick overview. Less fontsize -> more packages per display area, especially for big categories like network or community packages with 1200 / 2200 packages.
I understand that, but I find them hard to read without zooming (I'm on a 4:3 1280x1024 monitor, and I don't need glasses).
Although the network list is more readable even if it has same text size, maybe because it has more columns. Yeah the big white spaces in the middle of the page decrease readability for me.
Can you try increasing size of text in categories index and add a column? Or maybe/also waste some space by enabling again the sidebar/menu?
EDIT: also noticed the black bar on the top. I Like that. There are issues with black bar and black penguin skin in the icon. Might be worth it giving the wiki icon a white background, like a white circle.
I feel that you are trying to aim for unnecessary accuracy in the database. How often you have thought to refresh it to reflect all possible changes made to packages????
In my opinion it would be enough to use the "major" kernel version, meaning 4.4 at the moment as it reflects the packages on that. (targets still on 3.18 or 4.1 are more or less laggards.) 4.4 is supposedly the kernel version of the forthcoming release. (hopefully forthcoming)
Similarly, I am not sure if it makes sense to really comb through all platforms. You could just pick the 3-4 most popular platforms and use those as the reference benchmark about package availability. E.g. ar71xx (most popular target and largest number of routers), x86_64 to get all multimedia stuff etc., and then maybe mvebu and/or ipq806x (to get in the newest popular routers). I think that you would get over 95%, maybe over 99%, of the general package availability correct with just those platforms. (same way as packages are not equal, the targets' importance varies a lot)
For reference: Number of packages per architecture. Most of them being in the range of 3800 +/- 200
In order to avoid you going crazy %-) over the Table of Packages: Don't make it too complicated. As hnyman said: Focus on the latest supported kernel.
Adding new fields to the dataentries or multiplying dataentries with every new kernel is not what we want.
In the end, you always have to ask yourself if the amount of work you put into this really pays back, and if there are people out there really using all the extra info.
- Increased size to 160%, added one more column + re-added sidebar on https://wiki.lede-project.org/packages/index/start
- Increased size to 120% on e.g. https://wiki.lede-project.org/packages/index/network
Nice that someone noticed this little change
I now made the logo a bit larger, which helps a bit in recognizing the penguin.
[quote="hnyman, post:115, topic:123, full:true"]I feel that you are trying to aim for unnecessary accuracy in the database. How often you have thought to refresh it to reflect all possible changes made to packages????[/quote]I am aiming for a fire-and-forget solution.
The wiki server itself should run the script weekly to refresh the package lists.
Consider that the current package list was generated in like 30 minutes or so (thanks to multithreading) on a Xeon E3 1275 v2 and that I'm automating detection of what to work on by reading html pages as I'm not a fan of babysitting software.
For example, this line feeds the loop that cycles over package architectures (in the script the link is a $variable, I placed a link in this example so you can try it on your system and see the magic in action)
wget -q -O- "https://downloads.lede-project.org/snapshots/packages/" | sed '/^\(<a\)/!d' | awk -F\" '{print $2}' | sed s'/.$//'
And as you can see it is basically downloading a HTML page, piping that to the usual suspects that manipulate its text to get an arch list out of it.
You add stuff, you remove stuff, the script will auto-adjust. (There is also a tr at the end that turns newlines in spaces, but this stupid forum is deleting spaces so it is screwing it up and I removed it from the example)
Same for device archs/targets/subtargets (needed for some base packages and kmod-*). You add new stuff each day, once the buildbots compile it and it appears in download lists the next run of the script will load it in the lists here, no human intervention required.
I'm also using github's webpages and github's features to read "raw files" to download single makefiles straight off the source without using git (this is needed to read CATEGORY and SUBMENU from them). Git cannot let me download single files, Github lets me.
Next time I upload packages I'll use ssh so I will also test how the script runs on the server, (probably worse as it is a VM). I will probably need to limit multithreading a bit but that's easy.
[quote] In my opinion it would be enough to use the "major" kernel version, meaning 4.4 at the moment as it reflects the packages on that. (targets still on 3.18 or 4.1 are more or less laggards.) 4.4 is supposedly the kernel version of the forthcoming release. (hopefully forthcoming)[/quote]Yeah, would be cool to show who the laggards are though, as people might need that info from a package list.
(also for @tmomas ) I thought I can use the "kernel" dependency of these packages (entries in dependency list link to other packages already, and we have no "kernel" package anyway) as a link to a normal wiki page where we list what devices are on what kernel version (and place a N/A on version and architecture list in kmod packages, and delete the kernel version from dependencies so I don't need to duplicate packages anymore). The link would be using the datatable's linking using namespaces, same as with other dependencies.
I can easily generate the data for that by scanning the kmod package feeds (and I'm scanning the package feeds anyway), and then I can have the script generate a normal wiki page with a normal table (i.e. not using the datatable plugin, just standard wiki syntax) showing what targets are on what kernel version.
EDIT: here a kernel version wiki page with a table generated just now https://wiki.lede-project.org/playground/playground/table
[quote]Similarly, I am not sure if it makes sense to really comb through all platforms.[/quote] As said above, I don't really need to drop platforms, Performance with multithreading is acceptable and since each architecture is read by the same loop I don't care if it is running over 5 or over 50 architectures.
Performance without multihreading was garbage even for 1 platform, scanning the community package feeds the first time took ages.