remote users and LOCAL users - Thorsten?

Hi Thorsten

I would like to clarify a couple of points to make sure my understanding
is correct.

I assume that the use of a ServerEngine (*not^ the REMOTEServerEngine)
implies a Local (to the pc running the app) DB. This then effectively
means an embedded app in that there is no other access to THAT server
except from the app running the ServerEngine. To allow access from other
pcs needs some form of transport OTHER than NamedPipe or SharedMemory
transports and a RemoteServerEngine.

I want to have a user update a database via the net. I have setup remote
access to the pc via DynALias.com and have set the firewall to allow DB
acces etc, etc.

I would like (of course) to limit network traffic across the net and as
most of the update data will be invoice entry (ie no real interaction with
the remote DB) I want to have the user work "locally" until the pending
invoice data is ready to be sent (via the net) to the main database.

There will be an inital loading of other nxCachedDataSets for use as LOCAL
lookup tables, the user can then effectively work LOCALLY, only at the
competion of each invoce does the data need to be transferred.

I have an invoice entry table that is used for data entry and when
complete I use the copyrecords method to transfer these details to the
main table, then empty the local invoice entry table, ready for reuse.

I was contemplating a local invoice entry table actually (defined and)
stored on the LOCAL pc and access this via a LOCAL serverengine etc.

From previous discussions on nxCachedDataSet, it seems to me that I can
use this nxCachedDataSet feature to be the LOCAL table and then use the
CopyRecords method or the nxCachedDataSet.update method?

AFAIK the nxCachedDataSet.update method can ONLY update its own data (the
table from which the data has been cached), not update a different table
eg the Main Invoice table. Therefore the copyrecords method would need to
be used.

I hope this is understandable?

Thanks for any advice/assistance

--
Regards

Peter

Using Opera's revolutionary e-mail client: http://www.opera.com/m2/
Outgoing emails scanned by Trendmicro Internet Security

This might get a little bit longer...

To start with establishing some background knowledge.

First have a look here:

http://www.nexusdb.com/onlinemanualv...tml?client.htm
(mind the possible wrapping of the url)

As can be seen there, the componets you are used to work with,
TnxSession, TnxDatabase, TnxDataSet (TnxTable / TnxQuery /
TnxCachedDataSet / ...) are all wrappers around internal server
objects. If you Open a TnxSession component it means that the
TnxSession asks the server engine it's connected to for a
TnxAbstractSession object which is then stored inside the session
component. If you make calls against the TnxSession to add aliases, get
a list of aliases, add passwords to the session password list, and so
on, then all that TnxSession does is forward these calls to the
TnxAbstractSession. When you then open a TnxDatabase that database
first makes sure it's session is open and then calls a method on the
TnxAbstractSession to get back a TnxAbstractDatabase. Same for TnxTable
(which calls a method on TnxAbstractDatabase to get it's
TnxAbstractCursor) and TnxQuery (which calls a method on
TnxAbstractDatabase to get a TnxAbstractStatement when you Prepare it
and calls a method on that statement object to get a TnxAbstractCursor
when you open it).

The components in nxdb.pas know nothing about how to open table files,
execute sql, store data, build indices and so on. They are just
wrappers around these TnxAbstract* classes.

Now take a look here:

http://www.nexusdb.com/onlinemanualv...tml?nexusdbarc
hitecture.htm (mind the possible wrapping of the url)

That shows the differnces between remote and embedded server. What's
maybe not spelled out well enough there:

There is a unit (nxsdServerEngine.pas) that defines a
TnxBaseServerEngine an that defines these TnxAbstract* classes. The
reason why they are called so is because they are just that, largely
defined as virtual; abstract;. They do not contain any implementation.
So they don't know either how to store data, build indices or run sql
queries.

The main thing about the TnxBaseServerEngine is that it has a virtual
abstract method that returns a TnxAbstractSession.

TnxRemoteServerEngine and TnxServerEngine both derive from
TnxBaseServerEngine. Even tho TnxSession only knows about
TnxBaseServerEngine, if you give it a reference to a
TnxRemoteServerEngine or a TnxServerEngine it can call that virtual
method to get a TnxAbstractSession object.

Here is the twist now, it will never get a TnxAbstractSession. Instead
it gets a TnxRemoteSession or a TnxServerSession. (and the same for
Tnx*Database, Tnx*Cursor, Tnx*Statement, and so on).

The implementation of these two (which btw don't have any references to
each other) are wildly different. The only thing that the "remote"
classes know is how to write the passed in parameters into a message
and hand it off to the transport and how to get the out parameters and
result from the reply message that the transport gives back. So the
remote classes also have no idea whatsoever about how to store data,
build indices, .. you get the picture.

The TnxServer* classes on the other hand are the real deal. That's
where all the code lives that actually knows how to read and write
table files, how to maintain an index, how to execute sql. (It's
actually slightly more complicated with more levels of abstract base
classes and classes that implement the functionality, but that's not
really relevant for now.)

In summary, the only component that contains code to actually store
data (be it on disk or in an in-memory table) is TnxServerEngine.
Everything else just depends on TnxServerEngine to do that job.
Especially TnxCachedDataSet is just a TnxDataSet and as such just a
wrapper around a TnxAbstractCursor.

Whenever you are using a TnxSession, TnxDatabase, TnxTable and/or
TnxQuery there is ALWAYS a TnxServerEngine involved. Be it in the same
process or inside a different process with a TnxRemoteServerEngine and
Transports and Command Handler to facilitate communication to that
TnxServerEngine.

If you look here now you can see the multitude of different possible
configurations:

http://www.nexusdb.com/onlinemanualv...tml?deployment
options.htm (mind the possible wrapping of the url)

TnxSession and the other client components don't care if they are
connected to a TnxRemoteServerEngine or TnxServerEngine, all they know
about is TnxBaseServerEngine and the TnxAbstract* classes. And you can
have as many TnxSession's in your application as you like.

TnxServerEngine (and TnxRemoteServerEngine) don't care if the calls to
them come from a single TnxSession, many TnxSessions, a
TnxServerCommandHandler or direct calls. Or any combination of that.
And you can have as many TnxServerEngine's or TnxRemoteServerEngines in
your application as you like.

From these 2 simple principles the available configurations directly
follow. From having a single TnxSession connected to a single
TnxServerEngine or TnxRemoteServerEngine to having multiple TnxSessions
connected to multipe different TnxServerEngine's and
TnxRemoteServerEngine's and having the TnxServerEngine's available for
external access via transport and command handler at the same time.

Ok, that should give you enough background information.

Now to your particular scenario. In general I would strongly recommend
against making any database server directly available on the internet.
Be it NexusDB, MS SQL, Oracle, DB/2, Interbase or any other RDBMS.

A much better alternative would be to use something that's specifically
designed for this particular scenario, like RemObjects/DataAbstract,
RTC SDK, kbmMW, and other such solutions. They all have their
particular strengths and weaknesses and I can't really generally
recommend one above the other, it depends a lot on what your particular
requirements are.

Having said that, TnxCachedDataSet can indeed be used to work around
the main technical problem you would have with using NexusDB over a
high latency network like the internet, the "chatty" protocol between
client and server.

Like any TnxDataSet a TnxCachedDataSet needs to be connected to a
TnxDatabase which is connected to a TnxSession which is connected to a
server engine. As I already said above, it doesn't know anything about
how to store data. It's just a wrapper around a TnxAbstractCursor.

In addition to that it has a reference to a SourceDataSet. The source
dataset can (and usually should) belong to a different database (which
can be connected to a totally different server engine).

When you open a TnxCachedDataSet it first makes sure that it's source
dataset is open. Then it uses the the database it's connected to to
create a table with the same structure as the source dataset and opens
a cursor to that table. Batch reads and inserts are then used to
transfer the records from the source dataset into this new table as
efficiently as possible. The source dataset is now no longer relevant
and if it was opened by the cached dataset it's even closed again.

TnxCachedDataSet is now like any normal TnxTable just a wrapper around
a TnxAbstractCursor. If you assigned a value to the TableName property
then that's the name of the backing table in the TnxDatabase that the
cached dataset is connected to (if you didn't assign a tablename a
unique name was created internally). You can now even open that table
with other normal TnxTable's or use TnxQuery to run queries against it.

That so far is the basic design of the TnxCachedDataSet and it's what
the "cached" comes from.

An *optional* *addition* to this design was the later introduction of
update objects.

While there is only a single available update object currently,
TnxSqlUpdateObject, there is a general framework in place to write any
number of other update objects. Update objects (which can be referenced
from TnxCachedDataSets) get informed by the cached datasets referencing
them when they are opened. At which time the update object can create
an update handler that gets attached to the table that's backing the
cached dataset. The update handler is then informed about any insert,
update or delete taking place on that table (which means it will also
be aware of changes made through other TnxTable's or queries running
against that table) and can take appropiate actions.

For the current TnxSqlUpdateObject that means executing the specified
insert, update or delete query against the same database that the
source dataset is connected to. There is no "caching" taking place in
this particular update object.

As I just wrote this I realized one unnecessary restriction, and that
is that the queries of the update handler get executed against the same
database as the source dataset is connected to. I plan to add a
TargetDatabase enum property (with tdSameAsSourceDataSet,
tdSameAsCachedDataSet and tdSpecified as possible values) and a
Database property (which is used uf tdSpecified is selected) to
TnxSqlUpdateObject for the next release. That could then allow you to
have the TnxSqlUpdateObject insert records into a logging table
somewhere instead of executing queries that directly apply your changes
to the source dataset.

I hope this gives you enough information about what can and can not be
done with a TnxCachedDataSet.

Cheers,
Thorsten




Peter Sanders wrote:

> Hi Thorsten
>
> I would like to clarify a couple of points to make sure my
> understanding is correct.
>
> I assume that the use of a ServerEngine (*not^ the
> REMOTEServerEngine) implies a Local (to the pc running the app) DB.
> This then effectively means an embedded app in that there is no
> other access to THAT server except from the app running the
> ServerEngine. To allow access from other pcs needs some form of
> transport OTHER than NamedPipe or SharedMemory transports and a
> RemoteServerEngine.
>
> I want to have a user update a database via the net. I have setup
> remote access to the pc via DynALias.com and have set the firewall
> to allow DB acces etc, etc.
>
> I would like (of course) to limit network traffic across the net and
> as most of the update data will be invoice entry (ie no real
> interaction with the remote DB) I want to have the user work
> "locally" until the pending invoice data is ready to be sent (via
> the net) to the main database.
>
> There will be an inital loading of other nxCachedDataSets for use as
> LOCAL lookup tables, the user can then effectively work LOCALLY,
> only at the competion of each invoce does the data need to be
> transferred.
>
> I have an invoice entry table that is used for data entry and when
> complete I use the copyrecords method to transfer these details to
> the main table, then empty the local invoice entry table, ready for
> reuse.
>
> I was contemplating a local invoice entry table actually (defined
> and) stored on the LOCAL pc and access this via a LOCAL serverengine
> etc.
>
> From previous discussions on nxCachedDataSet, it seems to me that I
> can use this nxCachedDataSet feature to be the LOCAL table and then
> use the CopyRecords method or the nxCachedDataSet.update method?
>
> AFAIK the nxCachedDataSet.update method can ONLY update its own data
> (the table from which the data has been cached), not update a
> different table eg the Main Invoice table. Therefore the copyrecords
> method would need to be used.
>
> I hope this is understandable?
>
> Thanks for any advice/assistance



--

Hi Thorsten

Thank you for taking the time to write an extensive and very informative
reply.

On Mon, 05 Feb 2007 19:28:19 +0900, Thorsten Engler
<thorsten.engler@gmx.net> wrote:

> Now to your particular scenario. In general I would strongly recommend
> against making any database server directly available on the internet.
> Be it NexusDB, MS SQL, Oracle, DB/2, Interbase or any other RDBMS.
> A much better alternative would be to use something that's specifically
> designed for this particular scenario, like RemObjects/DataAbstract,
> RTC SDK, kbmMW, and other such solutions.

Other than the security aspect, is there any specific reason for the
"against" recomendation? Latemcy problems perhaps?

I anticipate the Dynalias connection will evolve into a VPN connection, as
this is a primary "test" and learning process (for me). Would you still
recommend something like the above when using a VPN?



--
Regards

Peter

Using Opera's revolutionary e-mail client: http://www.opera.com/m2/
Outgoing emails scanned by Trendmicro Internet Security

Peter Sanders wrote:

> Other than the security aspect, is there any specific reason for the
> "against" recomendation? Latemcy problems perhaps?
>
> I anticipate the Dynalias connection will evolve into a VPN
> connection, as this is a primary "test" and learning process (for
> me). Would you still recommend something like the above when using a
> VPN?

As VPNs usually doubles (or more) the ping times that would only
amplify the problem. So yes, using a VPN just strengthens my
recommendation to use a communication mechanism which is specifically
designed for this scenario.



--

Hi

On Tue, 06 Feb 2007 11:48:31 +0900, Thorsten Engler
<thorsten.engler@gmx.net> wrote:

> Peter Sanders wrote:
>
>> Other than the security aspect, is there any specific reason for the
>> "against" recomendation? Latemcy problems perhaps?
>>
>> I anticipate the Dynalias connection will evolve into a VPN
>> connection, as this is a primary "test" and learning process (for
>> me). Would you still recommend something like the above when using a
>> VPN?
>
> As VPNs usually doubles (or more) the ping times that would only
> amplify the problem. So yes, using a VPN just strengthens my
> recommendation to use a communication mechanism which is specifically
> designed for this scenario.

I was not aware of that, thanks for the info.

I have d/l the RTC SDK, the Tamaracka search I did after reading your
previous message provided many hits that praised the capabilities and
performance of RTC. I also read the reference to the input between
yourself and Daneijel of RTC. I was pleased to see the arrangements made
to enable NexusDB and RTC to work together.

I have not worked with this kind of process (RTC) before, but it seems
that it is relatively simple to learn the basics and have the initial
connections working ok. David Guest had an interesting message on how easy
it was to set up RTC.

I will see if I can follow the RTC examples and relate those to my needs.

Thanks again...

I might be back soon with more questions on RTC with NexusDB...


--
Regards

Peter

Using Opera's revolutionary e-mail client: http://www.opera.com/m2/
Outgoing emails scanned by Trendmicro Internet Security

Peter Sanders wrote:

> I was pleased to see the arrangements
> made to enable NexusDB and RTC to work together.
In all fairness, compareable arrangements exist with both RemObjects
and kbmMW.

> I have not worked with this kind of process (RTC) before, but it
> seems that it is relatively simple to learn the basics and have the
> initial connections working ok. David Guest had an interesting
> message on how easy it was to set up RTC.

One thing that RTC doesn't have currently as far as I'm aware (but
which is under development) is some form of integrated support for
streaming datasets and deltas. I'm not sure what the exact development
status is on that and if you need it you might want to ask about that.

For RemObjects there is DataAbstract which provides such functionality
and kbmMW has such functionallity too,

> I might be back soon with more questions on RTC with NexusDB...

I'm not really in a position to answer many technical questions about
RTC SDK (or RO/DA or kbmMW for that matter) as my personal experience
with all of these is limited to the work that was necessary to
integrate them with NexusDB.

Naturally I'll do my best to answer any NexusDB related questions you
might have.


--