nxRecover fails to recover...

Hi,

I got an "Block 28880 is identified as block 0 in the block header file"
error. I tried to recover it with nxRecover without luck, in the log there
is:
2010-08-30 19:14:51.088-02:00 6579466 5368 [$0/0] ERROR default
Trying to recover table C:\Delphi_addons\NexusDB2\Recover\CelGem.nx1

Howto recover everything or allmost everything ?

P. vd Lugt

On Mon, 30 Aug 2010 19:36:20 +0200, "PL"
<p.van.der.Lugt@acdbv.removethis.nl> wrote:

>Hi,
>
>I got an "Block 28880 is identified as block 0 in the block header file"
>error. I tried to recover it with nxRecover without luck, in the log there
>is:
>2010-08-30 19:14:51.088-02:00 6579466 5368 [$0/0] ERROR default
>Trying to recover table C:\Delphi_addons\NexusDB2\Recover\CelGem.nx1

If this is the only entry in the log file then the recover operation
was successful. This line is added to log the name of the table which
should be recovered.

Regards from Germany

Franz-Leo Chomse [NexusDB Expert]
franz.leo.chomse@ndx.nexusdb_x.com (please remove "_x" to reply)

Okay,

But the recovered records is only a small part of the total existing
records. Doing a recovery via EM gives also the same Block error.
I figured that the recovery stops at the faulty block and then exit ?

PL

"Franz-Leo Chomse [NDX]" <franz.leo.chomse@ndx.nexusdb.com> schreef in
bericht news:2p0o76l9mu2k3s3b863ju9nb4kr55jl8ng@4ax.com...
> On Mon, 30 Aug 2010 19:36:20 +0200, "PL"
> <p.van.der.Lugt@acdbv.removethis.nl> wrote:
>
>>Hi,
>>
>>I got an "Block 28880 is identified as block 0 in the block header file"
>>error. I tried to recover it with nxRecover without luck, in the log there
>>is:
>>2010-08-30 19:14:51.088-02:00 6579466 5368 [$0/0] ERROR default
>>Trying to recover table C:\Delphi_addons\NexusDB2\Recover\CelGem.nx1
>
> If this is the only entry in the log file then the recover operation
> was successful. This line is added to log the name of the table which
> should be recovered.
>
> Regards from Germany
>
> Franz-Leo Chomse [NexusDB Expert]
> franz.leo.chomse@ndx.nexusdb_x.com (please remove "_x" to reply)

PL wrote:

> Okay,
>
> But the recovered records is only a small part of the total existing records.
> Doing a recovery via EM gives also the same Block error. I figured that the
> recovery stops at the faulty block and then exit ?

No. Recover looks at all blocks and processes all blocks that match a certain
pattern (block signature ok, ThisBlock info ok, BlockOwnerID ok, ..) any block
that doesn't match is just skipped over.

If the recover only recovered a small part of the records you expected in the
file, then the file is seriously corrupted.

Also, "Block 28880 is identified as block 0 in the block header file" probably
means that this block (and potentially all following) is just filled with 0's
which would indicate a serious fault at file system layer or below (hardware).

I had a similar experience a long time ago. The recovery would work in
repairing the table, but it cannot restore all invalid data.

What I ended up doing was using the recovered and failed tables from the
recovery process to work out what record(s) contain the invalid data.
In my case there were things like datetime fields with 0.0 in it or
corrupted blob fields. I used EM to browse the table (via SQL statements).
The faulty records would cause EM to throw up AVs (and crash) when they
became visible on screen.

Once I figured out what record was the problem, I used SQL statements (which
don't require viewing the data to change) to either remove or repair the
corrupted field data for the specific record. E.g. Setting a bad datetime or
blob value to null (or some other legitimate value). Alternatively you can
delete the specific record and try to repack the table, which is only
successful when there are no table corruptions or invalid data.

After the table is fixed it is then up to you to track down in your app
where and how the corruption happened and attempt to reduce the chance of it
recurring. Not an easy task but you can narrow down your search through
deduction - focusing on areas of the app which modify, or save to, the field
that contained the bad data.

Make sure you keep a backup of the tables BEFORE attempting this method of
repair. It's a very manually intensive process and not an official NXDB
technique.

If you have a [daily] backup process for your tables, you can also transfer
[most of the] data from your backup database which may go missing after a
recovery.

HTH
--
Regards
Will Owyong [NDX]
Newcastle, NSW, AUSTRALIA
GMT+10

"PL" <p.van.der.Lugt@acdbv.removethis.nl> wrote in message
news:4c7c1111$1@DSVR011613....
> Okay,
>
> But the recovered records is only a small part of the total existing
> records. Doing a recovery via EM gives also the same Block error.
> I figured that the recovery stops at the faulty block and then exit ?
>
> PL
>
> "Franz-Leo Chomse [NDX]" <franz.leo.chomse@ndx.nexusdb.com> schreef in
> bericht news:2p0o76l9mu2k3s3b863ju9nb4kr55jl8ng@4ax.com...
>> On Mon, 30 Aug 2010 19:36:20 +0200, "PL"
>> <p.van.der.Lugt@acdbv.removethis.nl> wrote:
>>
>>>Hi,
>>>
>>>I got an "Block 28880 is identified as block 0 in the block header file"
>>>error. I tried to recover it with nxRecover without luck, in the log
>>>there
>>>is:
>>>2010-08-30 19:14:51.088-02:00 6579466 5368 [$0/0] ERROR default
>>>Trying to recover table C:\Delphi_addons\NexusDB2\Recover\CelGem.nx1
>>
>> If this is the only entry in the log file then the recover operation
>> was successful. This line is added to log the name of the table which
>> should be recovered.
>>
>> Regards from Germany
>>
>> Franz-Leo Chomse [NexusDB Expert]
>> franz.leo.chomse@ndx.nexusdb_x.com (please remove "_x" to reply)
>

Will Owyong [NDX] wrote:

> I had a similar experience a long time ago. The recovery would work in
> repairing the table, but it cannot restore all invalid data.

Yours and the original posters are totally different cases.

"Block 28880 is identified as block 0 in the block header file" clearly
indicates that there is a fundamental structural error in that file.

This is not an issue of some fields containing invalid values. This is a case
whole blocks being structurally broken.

Attempting to run any queries, select or update, against this table or using
TnxTable to access it will break down whenever any block with such corruption
is touched.

In your case, it was possible to get invalid values into fields because the
client is sending the whole record buffer to the server and the server simply
stores that. So if the client send a record buffer that it later can't
understand again this can result in client side errors.

In this case, the error is in data structures that are exclusively accessed
from deep inside the server engine. There is no way for any client side code to
affect this. And the error on access happens inside the server.

Furthermore, the code that generates the "Block 28880 is identified as block 0
in the block header file" error message is not just run after loading a block
in from disk. It is also run right *before writing* a block to disk.

That means any bug that might exist inside the server engine which could cause
this error (and there are no such bugs known) would never be able to actually
*write corrupted data to disk*. All these checks are performed as first step
during a commit before any data is written to disk.

The only way how this particular corruption can happen is somewhere outside
NexusDB, between the time that a (correct) block was written to disk by the
server engine and the time when the (now corrupted) block was read in from disk
by the server engine.

"Thorsten Engler [NDA]" <thorsten.engler@nexusdb.com> wrote in message
news:xn0gyj7jhx1w78004@news.nexusdb.com...
> Will Owyong [NDX] wrote:
>
>> I had a similar experience a long time ago. The recovery would work in
>> repairing the table, but it cannot restore all invalid data.
>
> Yours and the original posters are totally different cases.
>
> "Block 28880 is identified as block 0 in the block header file" clearly
> indicates that there is a fundamental structural error in that file.
>
> This is not an issue of some fields containing invalid values. This is a
> case
> whole blocks being structurally broken.
>
> Attempting to run any queries, select or update, against this table or
> using
> TnxTable to access it will break down whenever any block with such
> corruption
> is touched.

It is this point which I was taking advantage of when attempting to locate
the specific problem record(s). Where the TnxTable breaks down is where I
pinpointed the data corruption. My logic was that the block appeared broken
due to the invalid data offsetting/skewing the remaining block data
containing valid data.

>
> In your case, it was possible to get invalid values into fields because
> the
> client is sending the whole record buffer to the server and the server
> simply
> stores that. So if the client send a record buffer that it later can't
> understand again this can result in client side errors.
>
> In this case, the error is in data structures that are exclusively
> accessed
> from deep inside the server engine. There is no way for any client side
> code to
> affect this. And the error on access happens inside the server.
>
> Furthermore, the code that generates the "Block 28880 is identified as
> block 0
> in the block header file" error message is not just run after loading a
> block
> in from disk. It is also run right *before writing* a block to disk.

In my case, the same error message occurred, though with a different block
number. Hence the similarity. There was also significant record loss in the
recovery process which was unacceptable by my client which is why I had to
devise an alternative which preserved most of the valid records in the
corrupted block.

Unfortunately my case wasn't replicable but, as you've stated the issues are
completely different, I will not proceed with this issue and leave it to The
Masters <g>

>
> That means any bug that might exist inside the server engine which could
> cause
> this error (and there are no such bugs known) would never be able to
> actually
> *write corrupted data to disk*. All these checks are performed as first
> step
> during a commit before any data is written to disk.
>
> The only way how this particular corruption can happen is somewhere
> outside
> NexusDB, between the time that a (correct) block was written to disk by
> the
> server engine and the time when the (now corrupted) block was read in from
> disk
> by the server engine.
>

Thanks for your detailed explanation Thorsten. As always your insights and
perspective are greatly appreciated
--
Regards
Will Owyong [NDX]
Newcastle, NSW, AUSTRALIA
GMT+10