Persistence Layer

Launchpad suffers greatly by having no simple persistence layer. It has sql layer (Storm) but its regular business logic is intermingled with storage : we cannot test business logic without exercising the storage layer, we cannot test the storage layer without mixing in business logic tests, we cannot migrate components of our model to different storage systems (because of the intermingling). Performance suffers - there is no differentiation between object access and data retrieval, so any failure to preload data results in thousands of backend queries. And our test performance suffers because we're exercising the full stack every time we do anything. Lastly our business logic and storage logic have to change in lockstep because there is no separation between them.

As a Technical Architect
I want a dedicated, substitutable persistence layer
so that writing fast code, testing and evolution of our schema are easier to do than today.

This is not an ORM nor a separately library.

Rationale

Solving the performance problems endemic in Launchpad requires a code structure that fits the needs of our data storage layer; much of the Zope machinery we use is hostile to that structure : it assumes individual Python object and attribute access is cheap or free - which it isn't in our current conflated structure.

We need to achieve a high performance Launchpad to:

1. Satisfy the needs our users and stakeholders.
2. make good use of the substantial hardware investment made in running Launchpad

And we need to fix our testing story to be able to test much more rapidly than we do today.

The lack of persistence layer is one driving factor for the problems we suffer, and working around that adds significant burden to writing efficient code or tests.

Because of these considerations we're going to solve this infrastructure level issue now, so that we can (eventually) focus on interesting problems.

Stakeholders

1. Developers [folk that write code]

Constraints and Requirements

Must

What MUST the new behaviour provide?

Nice to have

Must not

What MUST it not do?

Subfeatures

Other LaunchpadEnhancementProposals that form a part of this one.

Workflows

What are the workflows for this feature? Even a short list can help you and others understand the scope of the change. Provide mockups for each workflow.

You do not have to get the mockups and workflows right at this point. In fact, it is better to have several alternatives, delaying deciding on the final set of workflows until the last responsible moment.

Success

Bugs are at: Bugs for this LEP link to a search for a bug tag or milestone. Use launchpad-project rather than a sub-project.

How will we know when we are done?

How will we measure how well we have done?

Thoughts?

Put everything else here. Better out than in.