All right, you guys, let's go ahead and get started.
I am back.
I know you guys all
missed me.
Just a couple of announcements.
Since you guys didn't have recitations this week.
I want to make sure that you guys remember the Design
Project 2 proposals are due next tomorrow
in class, as is Hands On 6.
Also, Quiz 2 is graded and is ready to be handed back.
If you go to office hours with your TA this afternoon,
you can pick it up or you can get it in class tomorrow.
We will post the statistics as soon as we get them.
We are still waiting to get the scores from one of the TAs
into the thing, so I do not want to say anything
until we know for sure.
All right.
What we have seen so far, we saw last time,
we talked about this notion of transactions,
we talked about the notion of making
actions durable and consistent.
This time what we are going to do
is look at a related topic, a slightly more advanced topic
that is more related to the notion of atomic actions
that we spent two or three lectures previous
to the one about transactions talking about.
If you remember, an atomic action,
we say it is both recoverable and isolated.
When we say an action is recoverable,
that means, remember, it either all happens
or it does not happen at all.
When we say an action is isolated,
it means, from the point of view of other actions
that are running concurrently in the system,
the action appears to only be one unit.
So then none of the intermediate states of the action
is visible to any other actions that are running concurrently.
And we talked about the use of the logging protocol
to allow us to recover actions.
We also, in the text and in class,
talked briefly about version histories as an alternative way
that we can recover actions.
And we talked about how to do isolation.
We talked about several different methods
for doing isolation.
We spent a while talking about locking as a mechanism
that we use to isolate actions from each other.
What we are going to talk about today is a related topic,
and it has to do with when you have actions
that are spread across multiple sites,
multiple different computers.
And this is going to tie together some of the concepts
that we learned in the previous section on networking
with the more recent stuff that we have been talking
about with atomic action.
The topic today is multi-site atomicity.
And just to give you a simple example of what
we mean by a situation in which you might care
about multi-site atomicity, suppose that you
are running a travel website.
You have some travel site, and that
travel site you have negotiated agreements
with various different people who sell airline tickets.
Maybe you have agreements with Jet Blue and USAir
and some other set of airlines.
And you want to make it so that when somebody purchases
a ticket, they may purchase a set of flights that
are purchased together as one unit where
there are different flights on different airlines.
I might want to purchase a flight from here
to San Francisco on JetBlue and then
a flight from San Francisco back to here on USAir.
Sometimes people want to use multiple different vendors
when they are purchasing their tickets.
And it may be the way that your travel site talks to JetBlue
and USAir is over the Internet.
They use some remote procedure call
to ask JetBlue or USAir to reserve a seat on their behalf.
But when somebody is using your website,
you want to create the illusion that these reservations that
are made on the website are sort of one atomic unit.
You do not want it to be that a person gets
a reservation on JetBlue and does not
get a reservation on USAir.
You want them to make a reservation
and get all of the reservation or none of the reservation.
In order to make that work, we are
going to need some sort of special support
from the travel site, JetBlue and USAir.
Suppose, for example, I did not have any special support
from the JetBlue website and I said
I want to purchase this ticket to San Francisco.
JetBlue goes ahead and says I am ready
and I purchased that ticket for you.
And then suppose we cannot get this reservation on USAir.
Now we are in trouble.
We need some way to back out of the action with JetBlue
and say wait, never mind, I did not
mean to actually purchase that ticket
because I could not complete the rest of my purchase.
We are going to talk about how to provide this kind of, what
we call, multi-site atomicity where
we want this whole action that includes purchases of tickets
from these different websites to appear as though it
was one atomic action.
Particularly we want to make sure that this whole thing is
both recoverable and isolated.
To sneak up on this topic, because there
are a bunch of different issues that are going to come up here
and are going to be a little bit complicated,
we are going to start by looking at a simpler
version of this problem.
Let's suppose that we have all of these things running
on the same computer together, that is they
are not connected over the Internet,
there is not this possibility of a message being lost like there
would be on the Internet.
So we are going to look at these things
as though they are all running together on the same computer.
In that case, we call these actions that
are running together on the same computer
"nested atomic actions".
Once we see how this works on just the single computer
case then we will sort of look and see
how it gets more complicated when we extend it out
to the multi-computer case.
Let's simplify this a little bit.
Suppose we had our buy ticket procedure looking something
like begin, buy on JetBlue, then buy on USAir and then end.
I am just going to call these two things A and B for now
just to sort of simplify it so I do not have to write JetBlue
and USAir over and over again.
And the property that we want is that each one of these actions,
in and of itself, should be atomic with respect
to the other actions.
We want atomic with respect to each other.
That is one property we want.
We want that because, for example, suppose
that buying JetBlue requires me to provide some credit card
number or debit card number that is going to go out and purchase
the ticket.
Well, I do not want to have the action that
purchases from JetBlue and the action that purchases
from USAir simultaneously decrementing my,
say, bank account.
Because we saw how you could imagine getting into trouble
where they would both read the balance at the same time
and then both decrement and then both write at the same time.
You could get some mixed up balance left in your bank
account, for example.
So if these things are, let's say, for example,
debiting a bank account, you want
to make sure that these actions are atomic with respect to one
another.
You also want to make sure that this whole thing is
atomic with respect to the outside world.
That is to say that the caller, somebody
who invokes this procedure to buy this pair of tickets
never gets to see a state where one of the tickets is purchased
and one of the tickets is not purchased.
It either looks like the ticket has been completely purchased
or it has not been purchased at all,
so we want it to be isolated.
And we also want it to be recoverable.
We want it to be the case that if we crash halfway
through here, after we have bought the ticket on JetBlue,
if the whole system crashes at that point, when
the system comes back up, we do not want the system
to be in some halfway state where
we paid the money for the JetBlue ticket
and have not paid the money for the USAir ticket.
So we should either complete the transaction
or we should completely roll back the transaction
and abort it.
So this is this notion of nested atomic actions, which
is we have this outer action and then it
has these two actions that are nested within it.
And each of these actions is, in and of itself,
an atomic action.
If you think about what's going on here for a minute,
so if you think about what the sort of condition that we want
is, we've got A and B.
And we want A to commit if B commits
and we want B to commit if A commits.
Because, if A doesn't commit, we need both of these actions
to definitely commit or definitely not commit.
Otherwise, we're kind of in trouble for this example.
But the way that I've worded this,
it kind of sounds impossible, like how
is it A is waiting for B to commit
and B is waiting for A to commit so how are we ever going
to make any progress?
And the trick is that we're going
to introduce a third-party, something that is
in charge of deciding whether or not the entire action has
committed.
So we're going to introduce a node S which
we call the "supervisor".
And S is in charge of deciding whether or not
the entire action, the action with both A and B inside of it
actually commits.
So let's see how that works because just seeing
that we still have to have some way of S
knowing that A is ready to commit
and S knowing that B is ready to commit.
And we still need some way of making it
so that A and B both make the decision to commit
at exactly the same time.
So let's see how we can go about doing that.
The idea is that we're going to introduce something
we call "tentative commits".
So what we want to do is get A and B
to a point where they're both exactly ready
to commit but they haven't yet actually committed
their results.
So they haven't actually made their results available,
but they want to give up control to S as to the instant
that they'll actually commit.
So what we're trying to get is a way
in which S can instantaneously make a decision
about whether both A and B commit or neither A
or B commits.
And so what we're going to do is,
the idea with a tentative commit is
we're going to run A and B until they tentatively commit.
And what that means is that they're going to do,
so we want A and B to do everything
except actually commit.
So what does that mean?
It means that A and B are going to read all of the data
that they would normally read, they're
going to write all the things that they would normally write.
If we are using a locking protocol they would acquire
all of the locks that they would need to acquire in order
to process the transaction, process
their part of the action.
But they're not actually going to commit.
So not committing means that, in particular, they
are not going to expose their results out
to the outside world.
So we say they don't expose results beyond S.
So a tentatively committed transaction,
I'm just going to write as TC, says
that it's not going to expose any
of the results of its action outside of S.
If we're using a locking protocol,
how do we prevent one of these nested actions
from being able to expose its results?
Or, how do we make it so that it doesn't expose its results
or it makes its results visible outside?
Yeah.
STUDENT: Whenever a stop action wants to commit,
we would just move the lock to its higher level of action,
to the action it belongs to.
Right.
That's essentially what we're going to want to do.
If we just want to make it so that the action's results
aren't visible, we're just going to make it
so that that action doesn't release any of its locks.
If it doesn't release its locks then
nobody else can get locks on any of the data that it updated.
And, therefore, none of its updates will be visible.
The solution that was proposed here
is what we're going to work up to,
which is that basically we want to make sure that if we want
S to be able to see the results of a tentatively
committed sub-action, which we will and I'll explain why.
Then what we're going to do is we're
going to hand the locks off from the sub-action
up to S, the superior action.
We're work through how that works.
The way that this is going to work,
this is going to allow us to get,
so we can draw a graph that looks like this.
We say S, we've got some action A, we've got some action B,
and we may, in principle, have other actions
that are sub-actions of these sub-actions.
And what we're going to do is we're
going to draw a graph where we put arrows
pointing from the sub-action up to its parent
action, the action that it is depending on.
And we're going to label the states,
we can label each one of these actions
in this graph as either tentatively committed.
Or, if it's not tentatively committed,
it hasn't finished doing all of its processing
yet we might label it as pending.
Let's look in a little bit more detail
about how we might actually get this tentative commit thing
to work.
And hopefully that will make it a little bit more
clear what's going on here.
And, in particular, what I want to do
is I want to look at the way in which a log
action on this machine might be being maintained.
And this is really going to help us get at how
we do recovery via logging.
This is for these nested actions.
Suppose we have some log and this has actions in it.
When S first starts running the transaction,
when the transaction first starts running this supervisor
module writes a begin transaction message.
And then what it's going to do is
it's going to invoke each one of these subatomic actions.
And each one of those subatomic actions
is going to write a begin record as well.
And then there's going to be some processing,
so these actions are going to execute,
they're going to obtain some locks
and they are going to update some data.
We're going to write those log records for that data that
was updated into the log.
And then, at some point later, we
will see tentative commits for A and tentative commits for B.
And then finally what we'll do is,
once those guys are tentatively committed,
we'll write the commit record for S.
Now let's look at what we can say at various points
sort of during it.
If you think of this log as being a timeline about when
things happen in the system, let's
look and see what we can say at various points.
One thing we can say is that this time when this commit
S record was written, this is the commit point
for this entire transaction.
So we say this is the commit point for both A and B
and anything else that maybe S did.
If you remember, what the commit point is,
it's sort of the point of no return.
Once we reach the commit point we've
guaranteed that this action is going to persist.
Even if the system crashes, it will recover in the state
as though that action had taken effect.
And if we crash prior to the commit point
then what we want to guaranty is that this action was not
visible.
When we recover we're going to undo any effects
that anything in this action did.
If we were to crash right here, just
before we wrote the commit record,
then we would undo this whole action.
And if we were to crash any time after we write
the commit record then we're going
to make sure that we redo any updates that the action may
have needed to do.
We're going to guaranty that this action is forced to disc.
But notice that there are these two tentative commit records.
What do these tentative commit records correspond to?
What these tentative commit records mean
is that A and B, this buy JetBlue and buy USAir,
did all of the work that they had to do.
So, in particular, it means that neither A or B is going
to have to acquire anymore locks,
they're not going to write anymore data.
They're at exactly this point where they're ready to commit.
And the thing that's going to make them commit
is the writing of this commit S log record.
Effectively, A and B, the sort of whether
or not A and B commit is now out of control of A and B
once they write this log record.
And it's completely in the hands of this outer supervisor module
S.
And so the outer supervisor module
S, of course it can commit, but you also
have to realize that it's also OK if this outer module aborts.
And if it aborts then it will write an abort log record
and we will have to go and undo the effects of the whole thing.
But we can still do that because we haven't, as of this point
here, actually exposed any results outside of this action.
So there is nobody else who has seen the effects of this thing.
We're still isolated with respect
to any outside action that might be running on the system.
So it's OK if we abort any time up to this commit record.
It also means that after this commit point,
this is sort of the point where we're going
to start exposing results.
If we're locking we're going to release our write locks
after this commit point.
And the act of releasing the locks
is what's going to make it so that other actions
outside of this system can see the effects of A and B running.
Just to make it clear, we say A and B commit or abort
when S commits or aborts.
I have just written CA here for commit or abort.
There is one other little detail that we
need to point out which is that S
can commit even if A or B fail.
So this may seem a little bit counterintuitive,
but the intuition here is that this action S,
suppose that S tries to run A and it cannot get a hold
of the tickets on JetBlue that it wanted.
Well, S is free to go and try and make a reservation
on some other airline that also satisfies the user's request.
So the fact that A failed doesn't say anything about
whether or not S is necessarily going to fail.
S still gets to decide whether it fails or not.
And this is kind of an important property
because it means that these actions that are running inside
of S are actually, while they are running,
are isolated with respect to S and the other actions that
are running on S.
Any of the updates that they make while they're running
aren't seen by S or any of the other actions.
This is just one little thing to keep in mind.
There is one last detail that I've sort of brushed over here
that we hinted at a little bit a minute ago.
And that's the problem which is what if A and B conflict
with each other?
We said that A and B might both update the same bank account
balance, right?
Well, we have a little bit of a problem
if that happens because I've got my S
and I've got my A and my B.
And it may be the case that suppose
we start running A first, A gets the lock on the bank account
and then B starts running and tries to get
the lock on the bank account.
And it waits for A because A is still holding
this lock on the bank account.
So we may have this situation where
B is waiting for A to release the lock on the bank account.
But the way that I've described this so far,
it may not be clear that B is ever
going to be able to actually obtain the lock.
Because we said that these tentatively committed actions
don't release their locks until after the commit point
has been reached.
It turns out that we need to sort of modify that statement
just a little bit.
And that kind of gets to the comment
that was made before which points this out.
What we want to say is that when an action like A
tentatively commits, we want to make it
so that S and its children can see A's updates.
After A has tentatively committed,
the other actions that are underneath S
should be able to see the updates that A made.
So A is going to go ahead and withdraw the money
from the bank account, and then it's going to say OK, I'm done,
I've withdrawn my money.
And now any other action within S
that needs to run that maybe also needs
to update the bank account will be allowed
to go ahead and do that.
So B could go ahead and run and update the bank account.
Notice that we haven't said that A's updates are
visible outside of S.
Nobody else gets to see that this bank account
balance got changed.
It's just the action B that gets to see that the bank account
balance is going to change.
Effectively, what this amounts to
is that when A tentatively commits
it assigns all of its locks up to the S action.
That's as much as I'm going to say
about this notion of nested atomic actions.
What this has given us is this way to have,
on a single site, one action that
is composed of multiple sub-actions
where those sub-actions are isolated from each other.
And all of the sub-actions either commit or abort
together as a batch.
But we said what we ultimately wanted
was the ability to do this across multiple sites.
Just to draw a simple architecture diagram
about the multi-site case seeing what this looks like,
suppose we have some action S and suppose
we still have our A and our B.
Now, just to conceptualize this, suppose
that there is some network in the middle of these things.
And this is a best-effort network
so it has all the problems that we talked about
that best-effort networks have.
It has congestion, it has delays, it can lose packets.
And the way to think about these things
is that these actions are going to interact with each other.
These nodes are going to interact with each other using
RPCs.
And they're just going to send actions,
they're just going to send requests
to each other and responses over these links.
So S is going to send RPC to A saying reserve a seat for me,
for example, and then A is going to send a reply
back saying OK, I went ahead and reserved that seat.
So what I want to do is sort of go
from this informal description of what
we want in the multi-site case to an actual description of how
the protocol works so you guys can see that there
are some pretty subtle details that are involved in this.
And it's worth pointing out that this situation is fairly
similar to many of the things that, some
of the problems that you're going
to have to deal with in the context of the design project
so it probably is a good idea for you to pay attention.
So let's talk about how this protocol would work.
The idea here is we want to provide --
Suppose we're still in this same travel site example.
The client wants to make these reservations over this network,
and the protocol we're going to use is a protocol called
"two-phase commit".
Suppose we have our node S which is the coordinator
node, the node that represents the travel site,
and we also have our two worker nodes that correspond
to JetBlue and USAir A and B.
I just want to show that each of these sites
is going to maintain a bit of a log that
reflects the state of the actions that it's running.
So I'm going to show the state of the log
on each of these nodes.
Suppose at some time S starts executing this action,
let's call it T, so it's going to write a log record that
says start T, and then it's going
to request that each of the subordinate nodes,
the sub-nodes goes ahead and does the processing,
say purchases the ticket that it wants.
It is going to send a message here saying, for example,
say the message consists of do something,
do X, purchase this ticket.
And, at the same time, it may also
send a message to B telling it to do something else.
Say, for example, do Y.
Now what's going to happen is that each of these guys
is going to receive this request to do something,
so it's going to write a log record that says start.
It's going to keep some information about what
it started doing, it's going to say X,
and it's going to remember that maybe this was
a part of transaction T.
And, similarly, this guy is going
to start Y which was a part of transaction T.
And then these two As and Bs now are just
going to start executing the action
that they were asked to execute.
So they are going to, for example, purchase the ticket.
They're going to run.
They're going to acquire whatever
locks that they need to process and then
they are, at some point, going to enter the tentatively
committed state.
The same thing is going to happen over here.
They are both going to run these actions.
When they reach the tentatively committed state,
what they're going to do is they're
going to send a message back that says something like did X?
This message is sometimes called a vote.
Basically it says whether or not this site
agrees that it was able to finish the work that it
was able to do.
So if it votes yes that means, yes, I'm done, I'm ready to go.
And if it votes no that means sorry,
I couldn't do the thing that you wanted.
So similarly B is going to send back its vote that says did Y.
So let's suppose, in both these cases, both of these actions
were able to do the work that they wanted to do.
At this point now what's going to happen
is that S is going to look at the votes from the actions
that it is tasked, and it's going to decide whether or not
this action is going to commit or is not going to commit.
So S is the one that's responsible for deciding
whether or not this entire action commits.
And suppose it decides it's going to commit,
it's going to write a record that
says commit this transaction T.
So this is now the commit point.
As soon as S writes the commit record, that
means this action is going to commit,
it's going to be made visible to the outside world,
all the work has been done.
And it's OK if it does this because A and B are
both in the tentatively committed state.
They've said I'm ready to go, I've
done all the work I need to do, I can commit whenever
you tell me it's OK to commit.
So, after this point, everything is going to commit.
The reason this is called the two-phase commit protocol,
typically this part is called phase one
where we're deciding whether or not we agree to commit.
And then here in this next step we enter phase two.
So what do we have to do in phase two?
Notice that these guys have tentatively committed.
A and B don't actually know whether or not
the action has committed so they don't actually
know whether they should release their locks
and make their updates visible to the outside world.
So we need to make sure that S tells A and B that OK,
this action is done, I've committed and it's OK for you
to also go ahead and commit and expose your results
to the outside world.
This is slightly different than the protocol
we looked at before because these things are
on different machines and so we have to pass information
from S to A and B to let them know that this action is
ready to go.
S is going to send a message to A saying commit,
A is going to write a log record that says commit,
and then it's going to do the same thing,
send the message to B that says commit.
And, similarly, B is going to write
a log record that says commit.
This is the basic two-phase commit protocol.
If you count the number of messages that you see here,
if you have N sites, the number of messages
you have to send in two-phase commit
is 3N in the basic protocol without any loss.
Notice I haven't said anything about what
happens when a message is lost.
And remember these best-effort networks
have this property that messages can be lost, we can lose data,
so we want to make sure that we understand
how this protocol works in the face of data being lost.
The other thing that we need to do
is to make sure that we understand
how this protocol works in the event
that either S crashes or A and B crash
at different points in the execution of the protocol.
So that's what we're going to talk through now, sort
of these nitty-gritty details about how we actually
get this thing to work in the face of these properties
that best-effort networks introduce.
To remind you guys how we deal with loss
in best-effort networks, I am just
going to very quickly review exactly
once RPC protocol that we talked about a few weeks ago.
Exactly once RPC remember is a way
to make it so that a procedure call gets executed once,
a remote procedure call gets executed once, and only once,
between a client and a server.
So if we've got our client and our server,
what we do is keep at the client,
we keep a list of messages, at the server
we keep a list of "nonces."
The client sends a request, a message, for example,
to the server asking it to do something
and it attaches a nonce to it, say N1.
So the client puts in its message table message,
N1 stores that information.
The server receives this request,
it stores the nonce in its nonce table and, one,
sends an acknowledgement and processes the request.
The acknowledgement comes back, and maybe it's
lost because these are best-ever networks.
Remember we have this timeout.
After some timeout period the client resends.
This is message, N1 again.
When the message gets resent, the server
checks to see if this message is already in its nonce table.
If it is, it doesn't process the message again but it sends
the [ACE?] for that message.
Now this [ACE?] message gets received,
the client crosses it off its message list because it knows
it's done processing.
That's the basic exactly once RPC protocol.
And what this guarantees is that this persistent client is
going to retry sending the request
until it gets an acknowledgement.
And the problem with that is that it can generate multiple,
the server can hear this message multiple times
so the server uses this nonce table to filter out
duplicate messages.
Let's see how we can use this notion of this exactly once
in our two-phase commit protocol.
I'm going to erase this and just redraw a similar example with
a [LOSI?] protocol instead of a [LOSLS?] protocol.
And just to sort of make the notation a little bit simpler,
let's now just suppose we have one worker site A.
We're not going to show A or B.
But this generalizes completely to as many As, Bs, Cs as we
want.
What's going to happen now is, what I want to do
is I want to keep a list of pending actions at S,
as well as the log of NS.
And I'm also at A going to keep a list of pending
actions and the log at A.
At some point S is going to go ahead and start
processing the transaction again and it's going to write
start T.
It's going to add T to its list of pending actions.
And then it's going to send this message that says do X to A.
Of course, it may be the case that this message gets lost
because this is a [LOSI?] network.
So we're just going to use our persistent.
We're going to make S persistently
retry so some time later, after some timeout,
it's going to resend this do X message.
And it knows that it needs to timeout
because it sees that there is this action here
that's still pending.
Now this site A is going to receive this request.
It's going to add X for transaction T
to its pending list, it's going to write its start
X of T record, and it's going to go ahead and process
just like it did before until it tentatively commits.
And now, once it is tentatively committed,
it's going to go ahead and mark this in its pending
table this action is tentatively committed
and it's going to send a request back that says did X.
Of course, this request can also be lost.
So, again, remember we have the server persistently retrying.
So, at some point, it didn't hear this did X request.
It's just going to say hey, do it again.
And now when A receives this request,
it's going to look up in its pending table,
it's going to see that it has already tentatively committed
this action.
So it's just going to not process the action at all,
it's just going to send back the request that says,
this should say do X, it's going to say did X.
Now, once S has received the tentative commits from all
of the other actions, it can go ahead and write
its commit record and we can go ahead and enter phase two,
just like we did before.
You can kind of see how this is going to work out.
The process is just going to continue in the same way.
After S writes its commit record,
it's going to go ahead and send the commit message.
And, of course, it is possible for the commit message
to be lost.
So, in this case, notice that the server doesn't actually
know whether the commit message has been lost or not.
And we haven't shown A sending out any responses
back to the server after the commit message has been sent.
So that means that A is the one that actually
has to retry in this case.
Now, A is just going to say hey, S, I did X.
And that's OK.
Now what's going to happen is S is
going to go ahead and look up in its table, when
it wrote the commit message to change
the state of this message to committed, this transaction
to committed, so S is just going to look up in its table,
see that the transaction is committed
and is going to send this message again.
And now, hopefully, this time it gets through.
And S can go ahead and change the state of this action
to committed, it can release its locks and the protocol is done.
One thing to note is that as soon
as A knows that the action is committed,
it doesn't need to keep any more state about the action
around anymore.
It knows it's committed.
That means that S has definitely heard about the fact
that it did X.
And so A can go ahead and just forget
any information it had in this pending transaction
table about the action.
We haven't quite solved the problem
but we still, in S, have to keep this information around
about the fact that the transaction committed
because we never actually know, in S,
whether this final commit message got through or not.
So it's always possible that A could re-request
the state of transaction T.
And S needs to be able to answer that correctly.
So this complicates this a little bit,
and there are a couple of solutions
that people have proposed.
The obvious one is we just add an extra round
of acknowledgements onto the end of this.
So that's a simple thing we can do, is just have A acknowledge
that it heard the message.
And then, as soon as S has heard acknowledgements
from all of its subordinates, it can go ahead
and delete the information.
There is another variant of this,
which is talked about a little bit in the text,
something called presumed commit where
basically if A doesn't know anything about the action,
it assumes that the action committed.
So A can discard the fact about any committed actions.
Getting that to work is a little bit trickier.
And the details of it are a little bit complicated
but you can sort of see the idea.
I just want to quickly spend a couple of minutes talking
about what happens in the case when these systems crash
during different phases of execution
so you guys can get a sense of how recovery
would work in this environment.
Let's suppose that S crashes.
And there are two situations we're worried about.
Either it crashes before commit or it crashes after commit.
If it crashes before commit that means
that, S is the sort of lead transaction here,
we want to treat this just like we
would treat this in sort of traditional recoverable
systems.
So if we crash before the main commit, well,
what we want to do is undo the effects of this transaction
completely.
So we're going to undo T.
Notice, however, that if there are multiple As,
Bs and Cs it may be the case that S crashes and some
of the subordinates are still processing messages and send
did finish processing requests to S.
That means when S crashes it recovers, it comes back up,
it undoes the transaction and it remembers that T aborted.
So it puts T in its transaction table
so that when it gets a request from somebody saying hey,
I finished doing this part of transaction T,
it can tell that guy oh, by the way, that transaction aborted.
Now, suppose that we crashed after the commit, well,
you know what's going to happen.
We want this transaction to be durable.
We said that the commit is the commit point,
we want this thing to appear to have happened.
So we need to make sure that we run redo on T
and that we remember that T committed.
Now, if A crashes, the situation is a little bit easier.
Basically, there are only two situations
we have to worry about in A.
It's either before or after we've
gone into the tentative commit state.
If it's before the tentative commit state,
well, we're just going to undo the effects of this transaction
completely.
We're going to roll it back and basically
going to forget about it.
And it's going to be S's responsibility
to try and redo this action with us, if it wants to,
or S may go ask somebody else to do this part of the action.
If we crashed after the tentative commit,
though, remember that at this point it's up to S.
This is A crashes.
At this point it's up to S.
So, after we've reached the tentative commit,
this action needs to go ahead and check with S
and see what the final outcome of this thing was.
It crashes, it comes back up, it sees
it was in the tentative commit state for T,
and so it sends a message to S saying
hey, whatever happened to T?
And then S sends a response back if it knows.
This is the basic outline of how we do crash recovery
and how we deal with lost messages
in this two-phase commit protocol.
With the last few minutes, I want to talk about something.
This two-phase commit protocol seems really great.
It seems like we got in this way we
have actions that are distributed
across multiple sites.
We've made it so that if the action commits,
we have this nice property that if the action commits,
we can make it so that either A or B don't commit
or they definitely both commit.
We have this way in which S is the ultimate arbiter
and authority of what commits.
This seems like a really nice system that we built.
And it is.
And it has all these great properties,
but there is one property that it doesn't have.
The question is, say in this environment with A and B,
do they make their results visible at the same time.
I had written commit before.
And, in some sense, they do commit at the same time
because they're going to definitely commit at the point
that this record gets written.
But the answer to the question are there results visible,
the question is are there results
visible at the same time, if you stare at this for a little
while you realize no because their results
become visible at the point at which these A and B receive
the commit message from S.
Not at the point that S writes the commit record.
So they're going to expose their results at a slightly different
point in time.
And, in fact, it could be a long time
because it may be that A crashed after it went
into the tentative commit record and it took it
two days to recover.
And then it came back up and then it checks with S
and says hey, whatever happened to T?
So it could be a very long time before, say,
A commits, makes its results visible,
whereas B may have done immediately
after it received the first commit message from S.
You might ask the question is it possible to guaranty
that A and B expose their results to the outside world
at exactly the same instant.
And the answer to this question, it turns out, is no.
And this is kind of a fundamental result.
The reason for this, let me just give you,
there's a simple example that's talked about in the book.
It's called the "two generals problem".
And the idea is as follows.
Suppose there are two generals and they have their armies.
They've flanking somebody who they are attacking
and they're on the sides of a valley
and they're both going to dive into the valley with the armies
at the same time.
They're trying to agree what time they're
going to do this at.
And they want to make sure they both attack at the same time
because, if they don't attack at the same time,
they're afraid that one of them will lose.
So the only way they have to communicate
with each other is by sending these messengers,
say, across the valleys.
They have some guy who runs across.
But, of course, this guy may not make it.
He may collapse from exhaustion or he may get shot or whatever.
So the first general sends a runner out
to the second general that says we'll attack at dawn.
And maybe that runner gets through.
And then the second general sends a runner back that
says yes, we'll attack at dawn.
And maybe that runner gets through, or maybe he doesn't.
The second runner doesn't get through,
and the first general says man, I
don't know if the second general heard about this or not.
So he sends another runner that says we'll attack at dawn.
And the second general says I already sent a runner
but I guess he didn't get through
and he sends another one back.
And, ultimately, they both need to agree
that they'll both attack at dawn so you
need a certain number of runners to successfully get
through in order for this to happen.
The issue here, though, is that suppose
the general have a fixed number of runners
and they want to say what's the maximum number of runners
that I could possibly ever need in order
to agree on this thing?
And, if you think about this for a minute,
you will see that there's no finite bound
on the number of runners that could possibly
be needed, because it's always the case that a huge number
of runners could be lost because this is
this best-effort network where there's always
some probability of something being lost.
There is always this infinitesimal little chance
that a million runners in a row wouldn't make it through.
So this is essentially the two generals problem.
And what it says is it's impossible for these two guys
to guaranty that they will achieve consensus
about something using a fixed number of messages,
using a fixed number of runners.
That means, in the case of two-phase commit,
what that suggests is that this S or one of these clients
may need to continually retransmit, say,
an infinite number of times, a very large number of times
before its request is actually processed.
Because there's always this sort of small chance of a message
being lost by the best-effort network.
So that's the two generals problem
and it's just sort of a nice result to keep in mind.
It's one of those results in computer science
that sometimes it turns out to be important.
In practice, in this kind of environment
it doesn't matter that much.
The probabilities of loss are small.
And so most of the time this is going
to achieve consensus after a limited number of messages.
So that's it for our discussion of fault tolerance
and recovery.
Next time we're going to start talking about security
and protection of information.
We will see you on Monday.