JeremyNicoll

Questions about rule management

Recommended Posts

Why do some entries in the supplied surf protection (aka Host Rules?) set have
action "Don't block"?   Why supply those rules at all?


When I export the host rules, the file created is named: a2user./dat/   But all the
other exports create .ini files.  Is that intentional?  I also note that some of the
.ini files contain only a timestamp if there's no content, whereas a2user.dat isn't
created at all if you have no user HRs.  Surely all this should be done the same way?


Within the exported host rules in a2user.dat there's a value, eg "1457125846" which
looks to me as if it might be a timestamp.  If it is, how does one convert it to a
form that makes sense, and what's it the time of?  Is it when the rule was created?
[i suppose the surf protection log shows when such a rule is actually used.]


Likewise the firewall and behaviour blocker logs show when parts of the application
rules are used.  If that info is only stored in the logs then I guess it's more or
less impossible to display when a rule was last created/modified, and when it was
last used, within the rule-editing dialogs?


The app rules dialogs allow one to make quite complicated sets of changes.  Is there
any possibility code could be added to log a rule's settings on entry to the editing
dialog and again on exit, so one could more easily keep track of what gets changed?    

Share this post


Link to post
Share on other sites

Why do some entries in the supplied surf protection (aka Host Rules?) set have action "Don't block"?   Why supply those rules at all?

It depends on your settings for the Surf Protection. For instance, if you have Privacy risks set to Don't block, then that will be reflected in the list of Surf Protection rules where it will say "Don't block" for any privacy risk hosts.

When I export the host rules, the file created is named: a2user./dat/   But all the other exports create .ini files.  Is that intentional?  I also note that some of the .ini files contain only a timestamp if there's no content, whereas a2user.dat isn't created at all if you have no user HRs.  Surely all this should be done the same way?

Host rules are stored in a different format from settings. INI files are actually not very efficient to process, and a lot of host rules could choke EIS if it stored then in an INI file.

The INI files always end in a timestamp, and thus there is always something in them. This is not the case for the custom host rules, where the file is completely empty when there's no custom host rules.

Within the exported host rules in a2user.dat there's a value, eg "1457125846" which looks to me as if it might be a timestamp.  If it is, how does one convert it to a form that makes sense, and what's it the time of?  Is it when the rule was created? [i suppose the surf protection log shows when such a rule is actually used.]

It is some sort of timestamp, however I'm not sure how to read/decode it.

Likewise the firewall and behaviour blocker logs show when parts of the application rules are used.  If that info is only stored in the logs then I guess it's more or less impossible to display when a rule was last created/modified, and when it was last used, within the rule-editing dialogs?

The only place I am aware of to see when a host rule was created is in the list of host rules in the Surf Protection settings. Note that dates/times are only displayed for custom host rules.

The app rules dialogs allow one to make quite complicated sets of changes.  Is there any possibility code could be added to log a rule's settings on entry to the editing dialog and again on exit, so one could more easily keep track of what gets changed?

We don't log when Application Rules are created/edited. It might be possible if we add another table to the local logs database, but of course this further increases the size of the logs, and with increased log size comes decreased performance. I'll pass this feature request along as well.

Share this post


Link to post
Share on other sites

It depends on your settings for the Surf Protection. For instance, if you have Privacy risks set to Don't block, then that will be reflected in the list of Surf Protection rules where it will say "Don't block" for any privacy risk hosts.

 

OK... and I see that if I add a rule of my own although it may also be set "Don't block"

and display in green like the supplied rules, at least its Category is listed as "My Own"

rather than "Malware hosts/Phishing hosts/PUP hosts/Privacy risks", which makes things

clearer.  Nevertheless it might be clearer still, if the action listed for one of these

supplied rules when it inherits a user-setting from the foot of the screen, said eg

   "Global user: Don't block"

and when a user sets one of their own rules it could say eg:

   "User custom: Don't block"

to emphasise the fact that all the former ones change if a user changes a global setting.

 

It is some sort of timestamp, however I'm not sure how to read/decode it.

 

I googled, which I suppose I should have done at the start, sorry.  I found a website

where one can plug a number like this in and get it converted, at:

   http://www.timestampconvert.com

[For anyone who's curious, the big number is basically the number of seconds since the

start (at 00:00:00) of 01 Jan 1970.  As there are 24 * 60 * 60 = 86400 secs in a day,

   1457125846 divided by 86400   is  16864 (days) plus 76246 seconds.

In a particular programming language I use a lot, I was able to calculate what day was

16864 days later than 01 Jan 1970... and it's 4th March 2016.

The remaining 76246 seconds is clearly nearly a whole day (as a day is 86400 secs).

76246 divided by 3600 (ie number of seconds in an hour)  is 21 remainder 646

&  646 divided by     60 (ie number of seconds in a minute) is 10 remainder 46

So 1457125846 represent local time   21:10:46  on  4th March 2016.] 

 

The only place I am aware of to see when a host rule was created is in the list of host rules in the Surf Protection settings. Note that dates/times are only displayed for custom host rules.

 

Presumably that's because regular updates from Emsisoft change the contents of the supplied rules on

a user's system, and there's no point in telling us when a particular set of rules first got a specific rule

added to it?

 

We don't log when Application Rules are created/edited. It might be possible if we add another table to the local logs database, but of course this further increases the size of the logs, and with increased log size comes decreased performance. I'll pass this feature request along as well.

 

I'm pretty sure SQLITE scales well, and unless one's storing millions of records, or maybe significantly more,

there won't be much performance impact increasing log sizes.   If that turns out not to be the case you could

always supply that log option with a default table size of 0 records, so only users who want the function could

enable it.

 

Share this post


Link to post
Share on other sites

...  Nevertheless it might be clearer still, if the action listed for one of these supplied rules when it inherits a user-setting from the foot of the screen, said eg

   "Global user: Don't block"

and when a user sets one of their own rules it could say eg:

   "User custom: Don't block"

to emphasise the fact that all the former ones change if a user changes a global setting.

I assume you mean in the notifications displayed when the Surf Protection blocks something?

I googled...For anyone who's curious, the big number is basically the number of seconds since the start (at 00:00:00) of 01 Jan 1970.  As there are 24 * 60 * 60 = 86400 secs in a day, ...

I was told earlier today that it is a Unix timestamp, although it sounds like you've already figured it out. ;)

Presumably that's because regular updates from Emsisoft change the contents of the supplied rules on a user's system, and there's no point in telling us when a particular set of rules first got a specific rule added to it?

We don't put the dates of when we add rules to our database in the actual database files, so there wouldn't be any dates/times for it to display for those.

I'm pretty sure SQLITE scales well, and unless one's storing millions of records, or maybe significantly more, there won't be much performance impact increasing log sizes.

It isn't an issue with the database itself, it's an issue with EIS parsing that many log entries. The more log entries it has to parse, the longer it takes. You'll usually only experience it as UI slowdowns or freezes, however it does also slow down the service starting, and thus slows down initialization of the Guards on startup.

Share this post


Link to post
Share on other sites
I assume you mean in the notifications displayed when the Surf Protection blocks something?

 

No, actually I meant on the Protection -> Surf Protection screen, where the list

of defined rules is displayed.

 

 

It isn't an issue with the database itself, it's an issue with EIS parsing that many log entries. The more log entries it has to parse, the longer it takes. You'll usually only experience it as UI slowdowns or freezes, however it does also slow down the service starting, and thus slows down initialization of the Guards on startup.

 

Except for when a user wants a log displayed, why would EIS have to parse log entries?

And if there's a problem when there are lots, surely it should only parse enough to

display a single screenful (and then either keep doing so in the background, or cease

until the user tries to scroll the display)?  But... parsing... these things things

should be trivial.  Is, by any chance, the cause of the slowdown that you repeatedly

ask SQLite for the next log record, rather than asking for a chunk (eg 100 at a time)

of them?

And, why would EIS need to reread old log records during startup?  That's not normal

in database processing.

Share this post


Link to post
Share on other sites

No, actually I meant on the Protection -> Surf Protection screen, where the list of defined rules is displayed.

OK, so you just want the list of host rules to say when you customized a built-in rule to change the default behavior.

Except for when a user wants a log displayed, why would EIS have to parse log entries? And if there's a problem when there are lots, surely it should only parse enough to display a single screenful (and then either keep doing so in the background, or cease until the user tries to scroll the display)?  But... parsing... these things things should be trivial.  Is, by any chance, the cause of the slowdown that you repeatedly ask SQLite for the next log record, rather than asking for a chunk (eg 100 at a time) of them?

And, why would EIS need to reread old log records during startup?  That's not normal in database processing.

To my knowledge, the entire contents of the SQLite database file are read when a2service.exe starts, and loaded into memory. I would believe that a2start.exe simply reads the log entries out of a2service.exe's memory in order to display them. In order to make scrolling of the logs faster, the entire log is loaded at once.

Share this post


Link to post
Share on other sites
OK, so you just want the list of host rules to say when you customized a built-in rule to change the default behavior.

 

I hadn't thought about that; my experiments so far had been when I added a

totally new rule.  I had the "Hide built-in list' option unselected & added

a rule for '0000000000000000000testjn.com', which obviously displays right

at the top of the list.

Suppose I add it with "Don't block" set.  Then in the whole list of rules the

"Don't block" text shown in green next to it looks identical to the "Don't

block" text shown in green next to sites which are in your supplied rules.

The point I was making is that the "Don't block" decision in the entries you

supply is different from the one that I added.  My one will remain at "Don't

block" even if I change the appropriate global option, whereas all the ones

in the supplied sites list will change.

If I edit one of your supplied rules for a site classed as a Privacy Risk,

changing the "Don't block" value it has (because that's the global setting),

to "Alert" and then back to "Don't block", it's again not entirely obvious

that that "Don't block" will remain set that way even if the global setting

is subsequently changed.  (Yes I know that it's clear that its now a rule

in 'My Own' set, with a date & time of change.)

All I was suggesting is that the values set because they came from global

settings should say that.

Meantime, I now have a Privacy Risk site whose entry used to say "Don't block"

because you supplied it tagged as a Privacy Risk.   How do I get back to the

original status where it exists just as a supplied rule?  If I delete it does

the next update reinstate it with your setting?

 

 

 

To my knowledge, the entire contents of the SQLite database file are read when a2service.exe starts, and loaded into memory. I would believe that a2start.exe simply reads the log entries out of a2service.exe's memory in order to display them. In order to make scrolling of the logs faster, the entire log is loaded at once.

 

If that's true, that the old log entries are cached in memory, I think that's

quite odd, considering that many users might want EIS' memory footprint to be

as low as possible.  I'd have thought that few people look often at the logs,

and when someone does they could live with a short delay while the first screenful

is parsed.

 

Share this post


Link to post
Share on other sites

Meantime, I now have a Privacy Risk site whose entry used to say "Don't block" because you supplied it tagged as a Privacy Risk.   How do I get back to the original status where it exists just as a supplied rule?  If I delete it does the next update reinstate it with your setting?

Yes, you can just delete the rule you edited, and it will return to the default.

If that's true, that the old log entries are cached in memory, I think that's quite odd, considering that many users might want EIS' memory footprint to be as low as possible.  I'd have thought that few people look often at the logs, and when someone does they could live with a short delay while the first screenful is parsed.

It's best to have the logs in memory so that they can be easily modified. You don't want to have to read the entire file just to write a single line to it, so you keep it in memory and when it get modified you only need to perform a write operation instead of a read and then a write with processing in the middle (which makes things very slow). Memory usage due to the logs isn't usually a problem, as even in extreme cases the logs can be moved to the pagefile if they are eating up too much memory and restored when they are being used. Granted, memory reducing techniques like that also cost a bit of performance. We do that with our database, and there's a setting to turn that optimization off due to the fact that it does reduce performance.

I know a lot of people prefer low RAM usage, however if you have plenty of RAM then you might as well use it rather than let Windows waste it all for caching/prefetching.

Share this post


Link to post
Share on other sites

> It's best to have the logs in memory so that they can be easily modified. You don't
> want to have to read the entire file just to write a single line to it, so you keep
> it in memory and when it get modified you only need to perform a write operation
> instead of a read and then a write with processing in the middle (which makes things
> very slow).

But... the logs are in an SQLite database.  Such databases can be vastly bigger than
the amount of RAM in a machine & one of the points of having a database represented
by a single file is that the SQLite DLL is entirely responsible for writing just the
changes to the on-disk file.  I can't imagine that any normal database application
"reads the whole file" anyway; what a waste of time that would be.
                                                               

Share this post


Link to post
Share on other sites

But... the logs are in an SQLite database.  Such databases can be vastly bigger than the amount of RAM in a machine & one of the points of having a database represented by a single file is that the SQLite DLL is entirely responsible for writing just the changes to the on-disk file.  I can't imagine that any normal database application "reads the whole file" anyway; what a waste of time that would be.

SQL database servers cache a certain amount of a database in RAM, which allows things to be loaded faster. Some database servers allow the amount of data that is cached in RAM to be configured, and some server administrators who have servers with enough RAM or small enough databases will set the cache size large enough for the entire database to be cached.

Regardless, in the case of SQLite there is no separate database server, so a2service handles reading the log data and caching it. With the default log entry limits, the log doesn't grow to a size that would pose a problem to load entirely into RAM. When I dump mine to a plain text SQL file, it is 1.02 MB, and it's using the default 300 entry limit.

Share this post


Link to post
Share on other sites

I still don't 'get' why you'd think SQLite would read or write the whole file just
to update a single record.  I'd expect that when it opens the database it would read
the control tables and then know where in the file the existing records are, and after
that do I/O just to the required blocks.   SQLite files don't normally grow & shrink,
unless one runs an explicit utility to do that, specifically because SQLite does its
own management of the layout of the data within them.  (At least as far as I was aware;
I've never used SQLite but I've been reading its support mail-list on & off for years.)

Share this post


Link to post
Share on other sites

SQLite isn't a database server, it's a format for storing databases in an SQL-like format in files, thus bypassing the need to have an actual database. It's fairly normal to read the entire contents of a file when you open it, especially if it's relatively small, and since the data in the log is (to my knowledge) cached in memory by a2service.exe I don't see why we would do it another way, but I can ask for confirmation.

Share this post


Link to post
Share on other sites

I know SQLite's not a server... but it doesn't "bypass the need to have an actual
database".  The logic in the SQLite DLLs and the user & control data in an SQLite
file certainly do between them comprise "a database".

I agree that in ordinary application programming it's common to read a whole file
in one operation - it's much more efficient than iterating through one in many I/O
operations... and subsequently write a whole file... but surely the whole point of
using SQLite is that the caller just asks for an SQLite connection to be made to a
database, and then asks for specific actions, eg "add a record to table x", and
doesn't itself do any IO.   SQLite does what it needs to do to maintain the file.

I'd be interested in confirmation, because I've come pretty close to asking specific
questions about this on the SQLite users mail list, since what you've said is so
much at odds with what I thought I understood about SQLite.  (The reason I've been
reading its mail list for so long is because I anticipate using it at some point
in the future, and hoped that lurking there would make the eventual learning curve
less steep.)

Share this post


Link to post
Share on other sites

Fabian: I realise that how SQLite works is irrelevant here, but how EIS logging works is not.

 

We got to here after I was told that increasing the log sizes would have a performance penalty,

and I was surprised by that (because of SQLite's ease of supporting huge databases.)

Share this post


Link to post
Share on other sites

> Fabian: We are not discussing...

If you don't mind me saying, that seems a rather terse response.

I'm not expecting to be told confidential matters or those that relate to the
security of the service that EIS provides.  But the original questions, about
the format of exported settings files, and the level of detail provided in
logs, matter to anyone who's concerned with managing change in rule sets.

As far as I'm concerned the contents of an /exported/ settings file, or a log,
are not "internal file formats".

I cannot imagine that I am the only such user who cares about these things.
Maybe it is rare amongst home users, but (based on my experience maintaing
mainframe systems for thousands of users) I would have thought your corporate
users would also welcome some improvements in logging (specifically logging the
before & after values of changed rules).   If that isn't done, then our only
alternative is either a tedious manual process of trying to document which
boxes got ticked/unticked every time someone changes something, or exporting
rules sets on a timed basis, maybe every hour, so that we can create our own
diffed lists of what changed in what time period.  I'm sure if you supported
this sort of thing better, you could use it as selling point.

It's useful to be able to pinpoint exactly what changed when, when looking
at other system problems.  It's the same reason that I turned on audit logging,
so the system security event logs record the creation & termination of every
process.  It means when weird errors happen and mention a pid, I can find out
what process that was and how it got started.

Share this post


Link to post
Share on other sites

We do consider exported rules an internal format. If we wanted it to be public, we would have documented it as well as the meaning of each field. If we start giving internal file formats out people will build tools or scripts working with these formats, which will cause issues if we ever want to change them, as we have to start considering third parties using those formats in unpredictable manners and we have to provide at least some degree of backwards compatibility which is something we simply do not want.

Bottom line: Everything that is outside of what is accessible via the GUI or command line tools is considered internal and not open for debate or discussion. If you want to write some custom script or tool that operates on those, we can't and won't stop you. But do not expect any help and most importantly expect those formats to change at any moment without further notice.

Share this post


Link to post
Share on other sites

OK, I understand the support nightmare for that from your point of view.

And I have no problem with the idea that the contents of exported files
may change format at no notice.  That doesn't prevent anyone from making
intelligent use of diffed export file sets.  After all, if an exported
file contains a section mentioning a rule with a name that /I/ assigned to
it, and what follows stays the same for days & days, and then changes, at
least that gives me a clue that something in that rule might have changed.

In any case, if you change a file format, there must be an indicator in
the files that says so, so that you can decide whether importing an older
format is safe or not.  Otherwise, why would a user ever bother exporting
a rule set?

Whenever I've written any code that runs through such a file I've always
done it with a state-transition parser and defensive coding so if a file
format changes my code is likely to notice.   But in any case I accept
that that's my problem, not yours.

Nevertheless I don't see why this should prevent your customers from asking
for changes, and it's hard to see how that can happen if you're disallowing
any discussion.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.