Toward a good "Flashing LEDE Instructions" page

As @tmomas pointed out in Creating Device Pages in LEDE Wiki, we need to find a way to encourage people not to re-write the flashing instructions, again, and again, and again, and again... Not only is it boring to read, but it introduces little errors that people propagate across Device Pages.

In that same note, I listed several reasons that I ignored (and hated) the OpenWrt "generic flashing" page. Let me re-cast those points as positive attributes for our new instructions page.

  • The new page should inspire confidence. Their first flash is a very anxious time for new users. They're about to take their functioning router and change it. There's a non-zero chance that they won't have a working router, so they need to be reassured.
  • The page should offer clear guidance for the most common cases. It definitely needs to address upgrading to LEDE from factory firmware. It should also cover upgrading from OpenWrt, or earlier LEDE firmware. Relegate other cases (FTP, TFTP, JTAG, mind-meld) to some other "Alternate Flashing Methods" page.
  • The page's heading should be a powerful statement that this is the way you do it. I propose "Standard LEDE Flashing Instructions"
  • The actual page name (in the URL) should appear authoritative. I propose that contain something like "StandardFlashingInstructions". (In OpenWrt, it's "generic.flashing", which always made me think that it was written in 10 minutes, and no one took the time to write a better one.)

Other factors that I'm not so sure of:

  • The page needs to address upgrading factory firmware vs upgrading LEDE/OpenWrt. It should refer people to both URLs in the dataentry for the firmware images.

  • I have the impression that some devices need different steps for flashing over factory firmware. For example, I have a vague understanding that some devices require you to install some funny firmware before you can flash OpenWrt. For these devices, the Device Page might need additional instructions.

[quote="richb-hanover, post:1, topic:52"]we need to find a way to encourage people not to re-write the flashing instructions, again, and again, and again, and again...
[/quote]My usual answer to such issues is "automation automation automation", fully mimicking Ballmer's "developers developers developers".

You make a template(s) with all the text and commands, then based on conditions and variables you assemble it in the device-specific page.

For example, for a factory update on a tp-link you can make a template with precise instructions, and call it in the device page of all tp-link devices, just changing the device and file names.
They should have the same webinterface after all.
You eventually end up with many template_tp-link_interface_2014-2017, template_tp-link_interface_2017-2020 as they change webinterface once in a while.

Or, for a device that has uboot bootloader you assemble the default "manual tftp flashing with uboot" template with device-specific flash offsets for firmware in the commands, and with brand-specific way of stopping autobooting.

This will probably require to add fields (again) to the toh, if the device can be flashed from factory, bootloader type (I think it's stated already), for flash offsets for flashing, and whatever else is required.

This system will also easily provide decent and updated links to the download binaries without human interaction (yes many OWRT wiki articles still link to random stuff in random places, that's bad).

I heard you prefer playing with templates on the wiki instead of editing, therefore I find some work for you. Happy? :stuck_out_tongue:

More seriously, I'm not that good with templates myself and I don't know how to have this happen in practice, so I'd rather have a more expert person do this while I'm mostly editing dumb documentation. Probably tmomas, that seems to be working on templates already from wiki logs.
But I can always learn to use templates, if needed.

[quote]The new page should inspire confidence. [/quote]Keeping all flashing instructions in a device-specific page is much better for this imho, as that's the device's page, people will trust more a device-specific page than a generic one. Because it's the page for their specific device, duh.

As a general rule of thumb, generic guides for everything are dev-only as the "genericness" does not inspire confidence and you can be sure that a generic guide WILL trick noobs as many won't be able to translate the generic instruction to a device-specific one.
I'd say to leverage the machines serving us already to also do the leg work here. Most edits on wikipedia are done by housekeeping bots after all, why can't we?

[quote]The page should offer clear guidance for the most common cases.[/quote]Using a template system should ensure that the common flashing instructions in each device page are all the same even if they are copied in zillions of different pages, as to change the tutorial you need to go and change the template page (that can be locked). Other flashing methods MIGHT have their template too if it makes sense, (like using UART/serial connection, that is required on a non-trivial amount of devices), but the others (JTAG, mind-meld) can be left to manual edits only.

Some specific devices have funky installation procedures, and there only way is manual edit, but as long as the procedure is the same on a class of devices (same brand maybe), you can make a template and use that. Like say Mikrotik_SPIflash_template.

As much as I would like this to be sufficient, I don't trust manufacturers/vendors to keep a consistent interface. A template for Vendor XYZ, model 123 might be correct, but if they release model 124, the GUI could totally change. Or worse, the XYZ/123v2 might be totally different.

And I don't know who would write each of these templates. (I think it would be madness to attempt it without having the device present...)

One (hopeful) thought: There are always two images for each model: factory and sysupgrade. Is it sufficient to state it this way?

a) If you're currently using vendor firmware, use LEDE factory image
b) If you're currently using anything else (LEDE, OpenWrt, etc), use sysupgrade

How could we determine if this is true? I wonder if a note to lede-dev would confirm this?

With "You eventually end up with many template tp-link-interface-2014-2017, template-tp-link-interface-2017-2020 as they change webinterface once in a while."

I meant that when the OEM decides to change the webinterface you make a new template and you use that from now on for that brand.

[quote]And I don't know who would write each of these templates. (I think it would be madness to attempt it without having the device present...)[/quote]If you mean for us now making the tutorials now, manuals or googling usually show the webinterface enough to do it right.
If you mean when adding a new device, the guy adding the device will have to either link in the right tutorial, or make the new template himself (if no such tutorial exists), or ask for help to us or the admin.

[quote]How could we determine if this is true? I wonder if a note to lede-dev would confirm this?
[/quote]I'm not a lede-dev, but I read the source of the image building process while trying to add a new device, and I used OWRT for a long time.

The "factory" image is an image that is disguised as a stock firmware, it contains all stupid dumb things that are required for the stock firmware to recognize the image as valid and flash it correctly. Only a small part of it is actually a flashable image (it is usually much larger than sysupgrade image too), and it may or not be in a form that is flashable (device-dependent) as it relies on stock firmware to process it first.

The "sysupgrade" image is a simple flashable image without any of that garbage, it can be flashed with the LEDE/OpenWRT updater system, but also manually (from bootloader, with SPI flashing tools, whatever), and it will work fine. Flashed it manually many times on many different devices.

In general I tend to agree with @bobafetthotmail here, a single generic howto flash page is likely not possible since procedures vary too much between models.

It should however be possible to break it down to a handful of different generic approaches;

  1. Devices which just expect an image upload via the OEM gui
  2. Devices which require CLI level access with or without serial
  3. Devices which require hardware modifications

Variants 2. and 3. are also divided into single- and multi-step installation procedures (like some require first loading an initramfs via TFTP, then actually write a sysupgrade image from within that - like when booting a live CD on PC to re-image a disk)

I would aim for the low-hanging fruit first and build a solid flashing instruction page for the OEM ui variant, providing a number, say 3, of screenshots showing a firmware update menu from some popular devices, like an Asus router, a Linksys router and a TP-Link device or something.

Would you think that this is a viable approach?

One other thing that I would like to see is the filenames for the correct images in the models/versions list (ie: for the TPLink 841 there are versions 1 through 11 listed in the TOH but the link for each one goes to the factory image and there is no filename, so good luck finding the upgrade version).

Aaron Z

Yes, I strongly believe that changes to the wiki pages should be done by someone who has personal knowledge of the device. If someone checks it out and writes the procedure in the wiki, they become the "world's leading expert" on that subject.

By this logic, there are way too many devices (~1,100 in the ToH) for this small team of people to tackle.
It's better to work hard to create a single, very good description that is true for virtually all devices.

PS I just saw a dissenting note from jow. I'll read and respond in a moment.

PPS See my next note that offers a possible solution for consideration

I realize that I never stated what I believe to be the Prime Directive for the first-time flashing page:

The Flashing Procedure Must Work First Time, Every Time

The procedure needs to optimize the customer's attention and brainpower. Because this is a (mostly) one-time operation, and because it's scary, they won't care if the procedure takes minutes longer than the "fastest possible". They are indifferent to the file size (as long as it's a few megabytes). They don't care about preserving settings, at least the first couple times. They won't even notice if there's an extra step or two that could be optimized out if they knew exactly what they're doing.

As long as there are a simple, finite number of steps that guarantee success (or something like 99% success), people will be happy following those steps.

If I'm reading @bobafetthotmail and @jow correctly, people could always use the "factory" image to flash their router.

  • It (obviously) is designed to work from stock firmware.
  • Is it true that you could flash the "factory" image on top of LEDE

If so, this is great news. Our "Standard Flashing Instructions" could say:

  • Download the "Factory" firmware using the link from your router's Device Page
  • Go to the "flash firmware" page of your router. Different vendors will have web GUI pages...
  • Follow the firmware flashing steps for your router.
  • Let the router reboot
  • Follow the rest of the steps (starting at step 2) at: https://wiki.lede-project.org/docs/quick-start-guide

Will this work for both stock/vendor firmware and LEDE? Thanks.

It would work for stock firmware. For flashing LEDE to LEDE you always need to use the sysupgrade image where available. In some rare cases both factory and sysupgrade images can be used but lets not distract people with such rare edge cases.

The procedure is always:

  1. Is the device running LEDE already? If not, advance to step 2 else jump to step 5.
  2. Is there a factory image available for your device? If yes, advance to step 3. else consult the device wiki page
  3. Is there a firmware upgrade menu available in the OEM web interface? If yes, advance to step 4. else consult the device wiki page
  4. Upload the factory image using the OEM web interface upgrade menu, goto quick start, installation is finished
  5. Is there a sysupgrade image available for your device? If yes, advance to step 6, else consult the device wiki page
  6. Is LuCI installed? If yes, upload the sysupgrade image using its upgrade menu, upgrade is finished. If no, advance to step 7.
  7. Is SSH reachable? If yes, scp the sysupgrade image to /tmp on the device and flash it using the sysupgrade command, upgrade is finished.
  8. Refer to debrick instructions

Thanks for the clarification. So the head of the Standard Flashing Instructions page would list these three choices:

  1. If your device is running stock/vendor firmware, use the "Installing LEDE on factory firmware" page
  2. If your device is running LEDE (and/or OpenWrt?), use the "Installing LEDE sysupgrade" page
  3. If you're not sure, please ask on the Installation Category of the forum. Be sure to include the brand, model, and version of your router and what information you see when you get to the configuration page.

Those two pages ("Installing LEDE on factory firmware" and "Installing LEDE sysupgrade") would then flesh out the remainder of the steps from above.

Is this correct? Thanks.

One additional piece of advice for the sysupgrade section/page that would increase the success rate:

It is recommended to flash without saving settings when switching to a different release or distribution (e.g. from a release to a buildbot snapshot, or from Openwrt to LEDE). That is especially important if you are changing from a newer build to an older release.

The user should re-build settings from scratch after flashing the router and getting it to sucessfully reboot with the default settings.

@richb-hanover - yes, this is correct.

Thanks, all, for this info. I will take 24h to pull together draft skeleton pages on the wiki that we can all tear apart :slight_smile:

[quote="richb-hanover, post:7, topic:52, full:true"]Yes, I strongly believe that changes to the wiki pages should be done by someone who has personal knowledge of the device. If someone checks it out and writes the procedure in the wiki, they become the "world's leading expert" on that subject. [/quote]I agree on this, but I'd like to point out that the device's web interfaces are usually exactly the same for all devices of the same brand because of obvious reasons (might have more or less pages for additional device-specific settings, but the "flash new firmware" page is part of the core that never changes unless the whole webinterface changes.

And they change the whole webinterface every 5 years or something, not every second weekend.

So to write a tutorial for using the webinterface to find the page where you can upload the file to flash you don't necessarily need to have the device on hand.

For more involved things, maybe yes, maybe not, depends from the info already in the wiki and that you find on the net.

[quote]By this logic, there are way too many devices (~1,100 in the ToH) for this small team of people to tackle. [/quote]Well, most installation instructions in the owrt wiki aren't wrong (they are just hard to understand, sometimes, or do some weird pointless things in addition), but yeah, for devices where there isn't an obvious installation instruction already I'd rarher not come up with anything and leave it empty for someone else to fill.

[quote]It's better to work hard to create a single, very good description that is true for virtually all devices.[/quote]imho, you will end with something that is too general to be really useful or inspire confidence.

I have created a draft Standard Flashing Instructions. This provides detail for Step 1 of the Quick Start Guide. In addition, I have created a sample Device Page that would tell people more about the flashing process (if the details are not covered in the Standard Flashing Instructions.)

Feel free to edit any of this. But please respect the Prime Directive (above - "The Flashing Procedure Must Work First Time, Every Time") in any of the changes.

That is, the listed steps are designed so the vast majority of people can follow them all the way to success. If there are odd cases or alternatives, they should be described on separate pages, with clear reasons that the reader would prefer to use the alternative. (For example, if you can't find a Device Page in the ToH, then it's reasonable to consider the "Alternative ways to find LEDE firmware images" link.)

Have fun!

Great work on the draft! Regarding the factory and sysupgrade substrings; this is almost always the cade with a few notable exceptions, eg. x86 or brcm47xx.

Thanks!

Is there a simple rule for those cases? (If there were just a few exceptions, we could conver this on the Device Page... But there are a lot of x86 and brcm47xx devices, so I don't think it would be a good mechanism...)

Or maybe it's not worth noting that URL would, as a general rule, contain 'factory' or 'sysupgrade'? I just threw that in there as 'belt and suspenders'. However, we're relying on people to follow these instructions carefully. They should be really be careful about the firmware image they download...

Update: I removed those "factory" and "sysupgrade" notes, since they wouldn't be true for large classes of devices, and they don't really make the process more reliable.

I think @jow might be referring to the fact you can use the installation ('factory') image to perform a sysupgrade as well on x86. That had me puzzled at first on my x86/64 APU2.

E.g. This is what you'd use for an ext4 rootfs with overlay, afaik:

https://downloads.lede-project.org/snapshots/targets/x86/64/lede-x86-64-combined-ext4.img.gz

Yes, I can see the confusion. I removed the comments (from the Flashing Instructions page) about 'factory' and 'sysupgrade' in the file name.

People rely on the Device Page, which ultimately gets its information from the 'dataentry' page that a person created when adding the device. Presumably, the original author enters the URLs correctly.

In cases where the names of factory/sysupgrade don't match a pattern (or, if they're the same in your case), the Device Page would be a good place to note that... (That is vastly simpler than trying to come up with a complicated algorithm for when a device has/doesn't have a particular string...)