JeremyNicoll

possible virtual memory problem with a2service.exe

Recommended Posts

> I've been working with computers since before IBM introduced the PC,

 

Me too, and for much of that time as an MVS (ie z/OS these days) systems programmer.   My view on "reinstall the app to fix a problem" would

be "raise a ticket because something in the vendor's app-update process corrupts the app".

 

>  ... Now test and see if the problem persists.   If it does now you can be sure it's EIS

 

Are you suggesting that when Task Manager and Process Hacker both attribute a climbing 'Private Bytes' value to EIS, that somehow they

have made a mistake?  If we don't trust the OS's own tracking of memory, should I also reinstall Windows?

Share this post


Link to post
Share on other sites

> I've been working with computers since before IBM introduced the PC,

 

Me too, and for much of that time as an MVS (ie z/OS these days) systems programmer.   My view on "reinstall the app to fix a problem" would

be "raise a ticket because something in the vendor's app-update process corrupts the app".

 

>  ... Now test and see if the problem persists.   If it does now you can be sure it's EIS

 

Are you suggesting that when Task Manager and Process Hacker both attribute a climbing 'Private Bytes' value to EIS, that somehow they

have made a mistake?  If we don't trust the OS's own tracking of memory, should I also reinstall Windows?

Hi Jeremy

 

Had a recent experience, where I was seeing a huge conflict involving 3 apps.  Everything, I mean everything pointed to EIS.  I finally sent logs to them, and the logs showed nothing.  Did some additional testing and keep getting the same results.   Finally I send all the relevant documentation to one of the other vendors.  He downloaded the two other programs and had a fix over night.  Wasn't EIS at all.

 

Kind of chuckled about the question about reinstalling windows.  I know quite a few people who would do exactly that.

 

Pete

Share this post


Link to post
Share on other sites

Apparently.   Unless vast numbers of your users reboot frequently, or have had the problem but don't realise it.

Based on the information you've posted here, we'd still be getting reports from users of random problems.

 

Why should it?  What's different about a fresh install compared with one that's been updated by your released updates?

Fresh configuration files, log files, re-registering the service and drivers, etc. The Factory Defaults button doesn't reset everything, and the only way to get the original config files back is to do a fresh install.

 

2.  Install a trial or Revo uninstaller and completely uninstall EIS

I generally recommend avoiding uninstall tools when uninstalling our software. They often miss things that Emsiclean would remove, and quite frankly if the uninstall doesn't get everything then you can unregister the drivers and service via the Command Prompt with a few quick commands and then delete the EIS folder.

... If we don't trust the OS's own tracking of memory, should I also reinstall Windows?

Windows doesn't exactly have the best virtual memory manager out there... ;)

Share this post


Link to post
Share on other sites

> Fresh configuration files, log files, re-registering the service and drivers, etc. The
> Factory Defaults button doesn't reset everything, and the only way to get the original
> config files back is to do a fresh install.

I sent you a copy of all the comfig files in use, on 21st April.  If there was a problem
in them wouldn't you have told me by now?  I have a problem believing that log files (ie
the SQLite database) could have anything in it causing such a growing memory use.

Is there more to registering services/drivers etc than a binary "it's registered or it
isn't" situation?  I mean, to the point that the product would run and appear to do
everything it's meant to, but still have a creeping memory problem?  Surely it's down
to how the app requests memory from the OS, and when it releases it?

Share this post


Link to post
Share on other sites

I don't see anything wrong with your config file, however just because it looks OK in Notepad++ doesn't mean that it is being processed correctly (there could be small issues that I am not seeing).

Is there more to registering services/drivers etc than a binary "it's registered or it isn't" situation?  I mean, to the point that the product would run and appear to do everything it's meant to, but still have a creeping memory problem?  Surely it's down to how the app requests memory from the OS, and when it releases it?

Drivers and services have options related to how they run. Those options are set when they are registered. Usually they can be changed later on, but some changes can be locked out (at least in the services manager).

Share this post


Link to post
Share on other sites

> I don't see anything wrong with your config file ...

You've passed the files to whoever's looking at the debug logs, though?


> Drivers and services have options related to how they run.  Those options are set when
> they are registered.  Usually they can be changed later on, but some changes can be
> locked out (at least in the services manager).

Have your developers ever seen instances of EIS drivers/services losing their original
setup, (on systems without active malware)?  I'd have thought that would be the kind of
thing that 'self-protection' would try to inhibit, and at the very least, identify if
it had happened.

Share this post


Link to post
Share on other sites

Has anyone looked at the logs yet?     Do they in fact trace memory being allocated and freed?  I did take a quick peek at them

before sending and didn't see any functions with the sorts of names (involving "mem", or "storage" or "free" etc) that looked as

if they might be relevant.  Of course I don't know what your functions would be named...

Share this post


Link to post
Share on other sites

You've passed the files to whoever's looking at the debug logs, though?

Not yet. They haven't asked to see it, but if they do then I will forward it to them.

Have your developers ever seen instances of EIS drivers/services losing their original setup, (on systems without active malware)?  I'd have thought that would be the kind of thing that 'self-protection' would try to inhibit, and at the very least, identify if it had happened.

It's possible for it to happen. We don't usually check the values of the registry entries that define the services and drivers, so we don't know for sure how often it happens (it's assume to be rare).

We don't prevent trusted applications from editing the registry. Obviously anything not automatically trusted would generate a Behavior Blocker alert, and can be blocked.

Share this post


Link to post
Share on other sites

I thought I should update you on the monitoring of memory use that I've been doing in the
last couple of weeks.  First, on 3rd May I updated to V11.7.0.6394.  In the following few
days I didn't use the machine very much, but also noticed what appeared to be less of a
memory problem; usage still climbed, but not so fast.   The Process-Hacker-reported private
bytes value for a2service climbed from around 500 MB when a2service first started, to about
1 GB, and then seemed to stabilise there for around a day and a half.

After that I had to do another restart when I uninstalled something, and in the course of
the next 24hr period a2service's private bytes went again from about 500 MB to 1 GB, then
very fast shot up to 2.1 GB.  In that period I'd done some quite heavy browsing - but there
was no sign of memory use being attributed to Firefox, but it does make we wonder if
certain types of website/webtraffic could make a2service 'work harder'?  (Flash is disabled
in my Firefox, so it can't be that.)

Since then I've tried to keep an eye on memory use to see if there's an obvious relationship
between what I'm doing, and rate of climb.  I've not spotted anything.

On Saturday 14th, it crossed my mind that it would be ironic if some interaction between
Process Hacker (which I've just updated from an older version to the current one) and EIS
could somehow cause EIS to grab memory & not release it... so after checking that the new
PH works, I removed its startup from my user-login process and rebooted.  Since then I've
been using Task Manager only to monitor memory use... and the problem is still with me.

Since that reboot, a2service's "commit size" (which is the same figure as PH's 'private
bytes') started off at 497,452 KB and has climbed to 1,239,620 KB.

I have also started running Sysinternals' "pslist -m" whenever the whim takes me, sending
its output to a date/time-stamped file, and keeping those.  The figures there do match
those that I see in TM.
 

Share this post


Link to post
Share on other sites

a2service.exe reduces memory usage (assuming you haven't turned the option off) by offloading unused parts of the database into the pagefile. As you use your computer, parts of that database will be needed, and Windows will automatically move them back into RAM as they are being used (this is why it causes a minor decrease in performance). Visiting websites of course will cause EIS to try to verify DNS requests against out Host Rules, which will cause part of all of our Host Rules to be moved back into RAM, thus at least temporarily increasing memory usage. Most browsers only request DNS info once per session, so if you visit a website then your browser will query DNS to get the IP address, and then it will cache it so that it doesn't need to query DNS for the same address again until you close the browser and reopen it. If you close your browser and reopen it frequently, or if you navigate to websites with different domain names frequently, then that could cause the memory usage to increase (at least temporarily).

Share this post


Link to post
Share on other sites

Are you seriously suggesting that having "Activate memory usage optimisation" selected would cause Private Bytes (PH) or 'commit size'

to increase from around 500 MB to (now) 1944 MB? 

 

Your own blog: http://blog.emsisoft.com/2016/04/13/why-antivirus-uses-so-much-ram-and-why-that-is-actually-a-good-thing/   says that the whole

signatures db is around 200 MB.  It'd need to be "moved back into RAM" seven times in its entirety to account for an extra 1400 or so MB.  

 

How big is the host rules part of the db?

 

Seeing as I /did/ provide debug logs, is anyone actually going to look at them?

Share this post


Link to post
Share on other sites

I've made some progress.

A long way further up this thread (eg post eight) I described how I was seeing very high use of the
non-paged pool of virtual memory.  This is kernel virtual memory which is always swapped-in, so
is in essence permanently allocated REAL memory.

I found a series of articles explaining ways of diagnosing memory leaks that allocate and then never
free such memory, starting here:

https://blogs.msdn.microsoft.com/ntdebugging/2012/07/31/troubleshooting-pool-leaks-part-1-perfmon/

In particular the second article:
https://blogs.msdn.microsoft.com/ntdebugging/2012/08/30/troubleshooting-pool-leaks-part-2-poolmon/

describes use of a tool named 'poolmon' which MS currently ship with their WDK (Driver Kit) as a
tool for people developing drivers. In essence, when drivers request this sort of memory, they are
meant to do so with a 4-byte 'tag' that's unique to that driver developer, and perhaps even to a
driver or even a particular part of the logic in one driver.  The poolmon tool lets you see the
kernel memory allocation in both the paged and non-paged pools, by tag.

I installed the WDK and ran poolmon, with an option that sorts its display by the number of bytes
of memory allocated under each tag.  As the screenshot shows, a bit more than 5 GB of REAL MEMORY
has been allocated to requests made under tag "WKMT".

This literal does not exist in any *.sys file inside C:\Windows  and its subdirectories.

It DOES exist in these four files:

  C:\Program Files\Emsisoft Internet Security\fwwfp32.sys
  C:\Program Files\Emsisoft Internet Security\fwwfp732.sys
  C:\Program Files\Emsisoft Internet Security\fwwfp64.sys
  C:\Program Files\Emsisoft Internet Security\fwwfp764.sys

This looks to me as if your drivers are causing this problem.

 

20160416 2150 08 (Poolmon screenshot on 20160518 0138).bmp

Share this post


Link to post
Share on other sites

Are you seriously suggesting that having "Activate memory usage optimisation" selected would cause Private Bytes (PH) or 'commit size' ™ to increase from around 500 MB to (now) 1944 MB?

No, I was simply explaining how memory usage can increase during certain activities when the memory usage optimization is turned on. Private Bytes, in my own testing, usually hover somewhere in between 500MB and 600MB (every time I check it's in the mid 500's).

How big is the host rules part of the db?

Compressed, it's about 2.2MB. Uncompressed, it's probably more than 50MB, however I don't have exact numbers. If I remember right, there are millions of domains and subdomains in the Host Rules, and I would believe we started compressing it because it was growing far too quickly.

Seeing as I /did/ provide debug logs, is anyone actually going to look at them?

The issue has been reported to our QA team. To my knowledge, a developer will be looking into this as soon as they can.

Share this post


Link to post
Share on other sites

> No, I was simply explaining how memory usage can increase during certain activities  ...
> Private Bytes, in my own testing, usually hover somewhere in between 500MB and 600MB

It's perfectly clear that not everyone has this problem.  Seeing over & over again that
your system doesn't have it, isn't helping finding out why my system does.


With regard to my discovery of the "WKMT" literal only in your fw... drivers, last night,
then I only searched the *.sys files in C:\Windows (and its subdirectories), and in
C:\Program Files\Emsisoft Internet Security\      but since then I've run a search across
all *.sys files on the disk... and still the only files containing that literal are the
Emsi ones.
        

Share this post


Link to post
Share on other sites

I haven't, because I don't see why the product installed and working fine, then updated automatically, should need it.

(And actually, I wasn't asked to do a reinstall.)

 

If a file was obviously corrupt or missing, fair enough...    If the developers think the code could have become corrupted

somehow, I'd rather someone made some effort to find out what is wrong, before a reinstall overwrites/resets anything.

Else, how will they ever find out how to stop such changes from breaking an install?

Share this post


Link to post
Share on other sites

I haven't, because I don't see why the product installed and working fine, then updated automatically, should need it.

It could fix the problem. If it doesn't, then you'll have ruled it out as a potential fix.

Share this post


Link to post
Share on other sites

If a new install fixes the problem, surely someone should be curious about why?    The updated product is meant to be the same

as a fresh install.   If it's not, surely someone at Emsi should be curious about what is wrong with the install.   So, if you'll tell me 

which bits of the registry, any files outwith   C:\Program Files\Emsisoft Internet Security   and anything else that I should backup

first, and then copy again after such an install, so that you guys can look at these things to see what has been changed, I suppose

I could try it.

 

I do understand the point that was made earlier, that any trusted application could have altered registry keys etc that might matter.

But I don't understand why, if that's the case, "self-protection" doesn't prevent that from happening.  

 

I have another product whose author was driven mad by (naive) users constantly complaining that a component had unregistered

itself.  Although he knows that the problem was users running registry cleaners and deleting eveything the cleaner suggested, and

explained this to such users, that did not stop the support requests.  Eventually he changed the product so that, in essence, it

reinstalls the licence registry keys every time it starts.

 

It seems to me that your product could, immediately after an install, store encrypted copies of critical registry data (if indeed this is

the problem) and - if not replace them, if the same entries are no longer present later on - at least log/alert saying that required

registry entries have been damaged.    

Share this post


Link to post
Share on other sites

Jeremy

 

There is an old saying in Texas here in the US.   Doesn't matter how many times you kick a dead horse.  It aint getting up. And that is what I see here.  You are trying to tell everyone what should be.  I have done many things to help developers (not just emsisoft) figure out problems., but one thing I wouldn't do is leave my systems on 24/7 for two weeks or more.  I shut them down every night. 

 

Also I have seen many software problems solved by a total uninstall, and reinstall.  Why?  Who knows, all I know is it works.  But something else I've learned working with quite a few companies over the years.  If you expect them to work to find reasons, and won't try the obvious, you probably won't get answers.

 

My advise.  Just uninstall, use the clean up tool, and reinstall, and see what happens.!!!!!

Share this post


Link to post
Share on other sites

The updated product is meant to be the same as a fresh install.

Actually, there's no point at which a fresh install will be the same as an existing install. Configuration files change, Application Rules are created, etc. Obviously we would expect the driver and executables files to be the same, but even that can't necessarily be guaranteed (if you need an example, then the System Restore makes a great one, as if will modify any drivers or services and delete our uninstaller).

So, if you'll tell me which bits of the registry, any files outwith   C:\Program Files\Emsisoft Internet Security   and anything else that I should backup first, and then copy again after such an install, so that you guys can look at these things to see what has been changed, I suppose I could try it.

The output of our diagnostic batch file should be enough (it includes the contents of registry keys), although you can save copies of all of your configuration files if you would like.

But I don't understand why, if that's the case, "self-protection" doesn't prevent that from happening.

The self-protection protects the running processes and the files in the product folder. It does not protect the registry keys. While I can't say for certain, I expect it's due to compatibility issues (imagine a system freeze in the middle of the System Restore trying to write registry changes while restoring to a restore point and being blocked due to self-protection). Since the Behavior Blocker prevents unknown applications from modifying drivers and services without approval, malware can't abuse this to disable our products. If a legitimate program does for any reason edit/damage registry entries for our products, then a reinstall should fix it.

... he changed the product so that, in essence, it reinstalls the licence registry keys every time it starts.

Our products will do that with drivers when the driver is not registered, and they will download missing files when running an update, however they don't spend a time verifying registry entries to make sure they haven't changed. The vast majority of the time such a thing is not an issue, and it would just cause boot delays and other performance issues for users.

Share this post


Link to post
Share on other sites

If I do a fresh install (of 11.7.0.6394) , and meantime you produce a new version, can I stop (eg by picking the 'delayed' update channel?)

my fresh install ffrom becoming the updated code?   With that choice will it still get signature updates etc?

Share this post


Link to post
Share on other sites

Yes, you can select the "Delayed" update feed, however please note that the Delayed feed will more than likely downgrade you to an older version of EIS.

The Delayed update feed only effects program updates, and database updates will be downloaded normally.

Share this post


Link to post
Share on other sites

I don't know; I've hit another bad patch health-wise and been using the machine much less in the

last few weeks.  I've also pre-emptively restarted it a lot more.  I have the impression that the

problem is still with me.  It was always clear that memory use could be stable for hours then shoot

up then be stable-ish again, ie it must have depended a bit on what I was using the machine for. 

Maybe whatever activity that was hasn't been so prevalent? 

 

Also, I seem to remember that I got bogged down in other problems when I did the reinstall you'd

asked for - not being able to persuade EIS that I was running in a private network.

 

I do not expect to have time to think about either of these issues until next week at the earliest.

Share this post


Link to post
Share on other sites

I do not expect to have time to think about either of these issues until next week at the earliest.

That's OK. Don't worry about checking until you're feeling up to it. ;)

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.