Replication component?

[Jesus Cea]

I am the maintainer of Berkeley DB backend for Durus.

The private branch of my code includes support for replication, between
other features. Now I am considering porting some of the code from the
"closed" version to the GPL one, and I think that replication could be
done trivially at Durus level: just call a function with the modified
objects pickls for each transaction. The function can do something so
simple as storing the pickles on disk, on a increasing filenumber name.

"Replaying" those pickles in order would replicate the storage.

I am doing just this when Berkeley DB replication is not applicable,
like when upgrading the backend storage format:

* Stop the program.

* Using ZFS (Solaris), I do an "instant" snapshot of the storage
database. This take a couple of seconds only.

* Configure the program to store the commit history (the pickles of
transactions).

* Launch the program again. It will keep the story.

* Now I clone the snapshot, aside, to be able to modify it. I run the
upgrade scripts there. This can take a lot of time (proportional to
database size). I have quite a few multi terabyte databases. This can be
hours. Or days. Or weeks.

* After I do the upgrade, I apply the pickles with the changes since the
snapshot, in order. This accoumulated data can be pretty large, so I
keep the application running and storing new pickles.

* When I finish the pickles backlog, I stop the application, apply the
last few pickles and swap the new database in place (ZFS magic).

* Launch the aplication again, running with an upgraded database, and
pickles logging disabled.

Would Durus developers consider this addition?.

PS: I do this using an "stacked" approach to Durus storage backends. It
is useful, for instance, for having a big read-only and small read-write
storages at once.

- --
Jesus Cea Avion _/_/ _/_/_/ _/_/_/
***@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/
jabber / xmpp:***@jabber.org _/_/ _/_/ _/_/_/_/_/
.. _/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz

[Binger David]

This sounds like a capability that would be useful to us, too.
When our site updates involve schema changes, such as a new
attribute added to a class, we update a copy of the database and
watch for new commits while the update is happening. If new
commits occur, we just start over. If not, we switch to using the
updated file and restart the application. Our downtime is just
a few seconds. If I understand your idea correctly, we could keep
a record of transactions that occur during the update and merge
them into the update as a final step. That would be nice for us,
especially if we ever have updates that take as much time as yours.

The Durus server does something like this for packing, except that
the list of accumulated records is held in memory instead of being
written to disk.

I'm not sure about the details. Why use separate files for all
of the pickles? I think we could use the existing FileStorage code
for this temporary store.

It seems like the essential function required is a command that
a client sends to the server to tell the server to start a transaction log.
Is that right?

- David

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I am the maintainer of Berkeley DB backend for Durus.
The private branch of my code includes support for replication, between
other features. Now I am considering porting some of the code from the
"closed" version to the GPL one, and I think that replication could be
done trivially at Durus level: just call a function with the modified
objects pickls for each transaction. The function can do something so
simple as storing the pickles on disk, on a increasing filenumber name.
"Replaying" those pickles in order would replicate the storage.
I am doing just this when Berkeley DB replication is not applicable,
* Stop the program.
* Using ZFS (Solaris), I do an "instant" snapshot of the storage
database. This take a couple of seconds only.
* Configure the program to store the commit history (the pickles of
transactions).
* Launch the program again. It will keep the story.
* Now I clone the snapshot, aside, to be able to modify it. I run the
upgrade scripts there. This can take a lot of time (proportional to
database size). I have quite a few multi terabyte databases. This can be
hours. Or days. Or weeks.
* After I do the upgrade, I apply the pickles with the changes since the
snapshot, in order. This accoumulated data can be pretty large, so I
keep the application running and storing new pickles.
* When I finish the pickles backlog, I stop the application, apply the
last few pickles and swap the new database in place (ZFS magic).
* Launch the aplication again, running with an upgraded database, and
pickles logging disabled.
Would Durus developers consider this addition?.
PS: I do this using an "stacked" approach to Durus storage backends. It
is useful, for instance, for having a big read-only and small read-write
storages at once.
- --
Jesus Cea Avion _/_/ _/_/_/ _/_/_/
... _/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQCVAwUBS5VJnJlgi5GaxT1NAQL8zQP7BL/PJC26TdY/KNldZ5m4b2OsDuplwxv8
ubxHywZlg7RWDRNDuTlCKk6Se9f3o76kk/jr4wvAX4qHF74I7A3UCsO/SoDQbHH5
VNU+KA9+jyZw6/MK7MNtCPAyt5YMw5JGCxnEoe+weoYmTCQB0s2TAdfQaDAwRvd6
4cdkHoHk4qQ=
=SHUQ
-----END PGP SIGNATURE-----
_______________________________________________
Durus-users mailing list
http://mail.mems-exchange.org/mailman/listinfo/durus-users

[Jesus Cea]

Post by Binger David
This sounds like a capability that would be useful to us, too.

Good!.

Post by Binger David
I'm not sure about the details. Why use separate files for all
of the pickles? I think we could use the existing FileStorage code
for this temporary store.

Well, it is just how my current patch works. Dirty and ugly, but it
works. It is easy and allows easy replication to a remote site, for
instance.

Post by Binger David
It seems like the essential function required is a command that
a client sends to the server to tell the server to start a transaction log.
Is that right?

I use stacked backend storages, as I said. Not sure if allowing a client
to activate it on demand is appropiate.

Using stacked storages allows to be more flexible, because the stacked
component can do anything with the transactions. They can be stored on
disk, or they can be sent to a remote site for replication. Or simply
extract statistics. Or deploy the readonly+readwrite backends I talked
about in the last message.

Not sure a client can demand this kind of things on the fly, using the
durus wire protocol. Seems more a thing you should do instancing the
right backends in the server process.

- --
Jesus Cea Avion _/_/ _/_/_/ _/_/_/
***@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/
jabber / xmpp:***@jabber.org _/_/ _/_/ _/_/_/_/_/
.. _/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz

[Binger David]

Post by Jesus Cea

Post by Binger David
This sounds like a capability that would be useful to us, too.

To be clear, the capability I mean is the capability to execute long updates
on huge databases without stopping the service during the update.
I'd like to have this capability with very little or no change to the
Durus storage code, and that seems possible.

Post by Jesus Cea
Good!.

Post by Binger David
I'm not sure about the details. Why use separate files for all
of the pickles? I think we could use the existing FileStorage code
for this temporary store.

Well, it is just how my current patch works. Dirty and ugly, but it
works. It is easy and allows easy replication to a remote site, for
instance.

Post by Binger David
It seems like the essential function required is a command that
a client sends to the server to tell the server to start a transaction log.
Is that right?

I use stacked backend storages, as I said. Not sure if allowing a client
to activate it on demand is appropiate.

I'm not sure why. Clients do anything else.

Post by Jesus Cea
Using stacked storages allows to be more flexible, because the stacked
component can do anything with the transactions. They can be stored on
disk, or they can be sent to a remote site for replication. Or simply
extract statistics. Or deploy the readonly+readwrite backends I talked
about in the last message.
Not sure a client can demand this kind of things on the fly, using the
durus wire protocol. Seems more a thing you should do instancing the
right backends in the server process.

Your interest in backend flexibility is reasonable, but I am not so interested
in that. I don't see any obstacles to using clients for replication or running
statistics, no matter what backend you use.