Indeed, added to client agenda!
If you could give a quick intro/demo to rest of team there it would be really useful.

So all update modules should accept the -u flag (update)

Not in general, I would say. This is one of the flags I have renamed in the module-image command, to -f-, or -file. The reason is that -u isn't a good generic name, since the payload can be absolutely anything, an image, a file with link, a file with a Docker SHA, a key to be used somewhere, a script, etc. It can also be used multiple times, depending on what the module needs, and not all the files are necessarily updates. So I thought "file" was more generic ("data" could be a candidate as well, which is what is used in the format).

But yes, some others are truly generic, such as -o, and -t.

However I think we should assume that the "full/global" list of CLI options is only offered by mender-artifact itself (e.g. -n , -v, -s). Otherwise the update modules would all need to document and implement all of those. So this aligns with the "add to Artifact" approach for update modules.

Maybe we're getting a bit too much into the details as you say, since this is mostly an overall technical design problem, and not so much about the individual flags. The problem is in the passing of flags from mender-artifact, into the called update module, and then again from there into mender-artifact in the sub invocation. One possible approach would be something like:

In other words "save" the flags in the environment. This method feels a bit hacky to me, but it does produce a nice end result. One problem I see is that we could never add any flags to mender-artifact, because they would risk overlapping with the flags from an update module developed before the flag was added, and would therefore hide the flag from it. This is another reason why I think merging of the flags we support, and the flags they support, isn't a good idea.

It feels more and more like this is a topic for the client meeting though.

Overall I think this is looking great, I think we are getting a bit confused due to the level of detail discussed here.

The purpose of the naming prefix (e.g. mender-artifact-module-deb) and standardized minimal CLI options was for the end user to be able to use mender-artifact and mender-artifact-module-deb seamlessly. What would be very confusing is if mender-artifact-module-deb would implement a different set of CLI flags than mender-artifact, or worse, it would use the same CLI flags but they would mean something else (e.g. it implements -n as "remove Artifact", instead of "Artifact name").

But to be honest I have not mapped out what those CLI options would be. Which ones do mender-artifact-module-deb need? I suppose list of deb files to be built at least? So all update modules should accept the -u flag (update). Then we also have the output side -o (output-path). Maybe those are the only minimal CLI option they should accept? They can accept more, if needed (e.g. if delta updates has some special input/config). However I think we should assume that the "full/global" list of CLI options is only offered by mender-artifact itself (e.g. -n , -v, -s). Otherwise the update modules would all need to document and implement all of those. So this aligns with the "add to Artifact" approach for update modules.

If we just have a policy for it that is good enough for now, I think.

I think so.

So to summarize: We should focus on adding to existing artifacts, in other words a workflow where you add something to an artifact, little by little, and we should allow the global parameters to be set both early and late. That makes sense, and is probably easier to wrap one's head around, indeed. Instinctively, I think that adding global parameters late will probably be more work than adding them early, due to the already existing rigorous checking of artifact validity during reading. But I'll evaluate this as I go, and if it becomes significant I'll create a task for a later iteration.

For clarity I would also require some naming prefix for the update module builders, such as "mender-artifact-module-deb" (instead of "deb-update-module"), and they must implement the same set of CLI options (minimal set required, they can have more if they want).

I agree with the reasoning, but I think it will be hard to enforce this, since ultimately it's up to the update module to decide how to call the build command. We could get around the naming convention by using mender-artifact as a thin calling mechanism, like:

However, I'm skeptical about enforcing any flags here. If the module will call mender-artifact internally to do the build, how will we enforce that the update module uses the flags when doing so? I don't see a sensible way of solving this on the technical side, so maybe we should rather focus on best practice and encourage that all module writers pass all unused parameters down into the mender-artifact command. For example the module would consume the -x flag, but would pass through -n unharmed into mender-artifact, since it doesn't use it itself. Does this sound ok, or do you have a different idea for how the build command should be invoked?

OK. To be honest, I don't think it would be confusing to have a parameter to add updates (e.g. mydebs.tar) to an Artifact. If you are working with multi-update Artifact then you should already understand that there logically are multiple updates inside.

Interface/CLI aside, I would just question where my state scripts, device types, etc. should be added when working with "partial" Artifacts and merging them. My expectation would align with what you are explaining above (device type would be intersection, depends/provides union, etc). However, I think this is a quite complicated workflow and hard to wrap my head around what the "end" Artifact would be and behave like. And for this reason I don't think users would want to work this way; they would likely add all the "general" attributes first or (more likely) last.

Since you think it would be the best way to work with complete Artifacts and it is less work, let's continue this way. You are also right that for delta updates, only the update module would know about these generic attributes (depends/provides), and the delta update module builder should own setting those (which are in fact, general Artifact attributes).

However, let's think through the desired workflow for going from no Artifact and two updates into a full Artifact. At a high level, I think this would be:

• Add first update (creates new Artifact if none exist)

• Add second update (adds to Artifact)

• ...

• Add last update

• Set final meta-data like device type, name, etc.

• Sign the update

So I would avoid working with the required "top level" meta-data in the intermediate steps here. I also think we need an option to add to an existing Artifact, as generally the intermediate Artifacts would be useless I believe (or at least quite rare - if you are testing an Artifact before deployment I think you want to have it all so your test is the same as prod). I think your "merge" command could simply merge into an existing Artifact (destination input).

There are more details to figure out here, like you say, what should the device type and name be until the user sets the "final" ones? Auto-generated is probably fine, we could even make them invalid (so they cannot be deployed until the user sets the final ones).

For clarity I would also require some naming prefix for the update module builders, such as "mender-artifact-module-deb" (instead of "deb-update-module"), and they must implement the same set of CLI options (minimal set required, they can have more if they want).

In conclusion I think we should make a more clear distinction between "in progress of building" Artifacts and "final" Artifacts.

Does that make sense?