Lots of Files in Temp Dir hold open by a2guard

[iammike]

With the new version (10.0.0.5361) I am seeing that a2guard is creating files (with random names 32 chars in length and ending with .tmp) in the users temp dir which are locked.

 

Everytime I open EAM it's creating more and more (at the moment 26 already) and the filesize is increasing.

 

Why is this ?

 

Edit: When Closing EAM "Security Overview" Window, the number of files that are locked go down to 11.

Edited August 16, 2015 by iammike

[JeremyNicoll]

I just came to the forum to ask exactly the same thing.  If EAM must create these things, could it please do it in an Emsi subdirectory of %TEMP% so that they are easier to ignore?

[stapp]

I can confirm everything that you see. 

 

Let's wait for a dev to comment  

[Fabian Wosar]

Those files are created by the error reporting component we use and nothing to worry about.

[iammike]

Those files are created by the error reporting component we use and nothing to worry about.

Oke, but can you do change it that it logs to a Subdirectory as "JeremyNicoll" suggests to make it obvious that the files are created by EAM ?

 

<snip> If EAM must create these things, could it please do it in an Emsi subdirectory of %TEMP% so that they are easier to ignore?

[Fabian Wosar]

That is unfortunately not possible at the moment. 

[iammike]

Oke, but I may I then ask if you can change the naming to for example:

Emsisoft<sid> or

EAM<sid>

[Fabian Wosar]

No, we can't do that either. This is a third party component and changing file names as well as the directory they are stored in is not a configurable option. I have asked them to add these features in the future. But until they eventually do, we can't change them.

[JeremyNicoll]

In that case it's time you put pressure on the 3rd party.  It's not acceptable (to me) that your app - which never did this in the past - suddenly puts lots of anonymous files into TEMP.  It makes management of these files pretty much impossible.

[Fabian Wosar]

We do use a whole lot of "anonymous temporary files" and always have. Especially when unpacking archives during scans. I guess you simply didn't notice them before since unlike these new files they aren't there all the time. That being said, we decided to roll back to an older version for a different reason. So at least for the immediate future you won't have to deal with those temporary files anymore.

[JeremyNicoll]

I have no problem with an app creating a temporary file, using it, and then deleting it.  But - for me - any app that creates files which hang around for long periods of time should ideally do so in a way that identifies what created it.  I really dislike apps that don't clean up temporary files after the app ends.   Many users, especially naive ones, end up with thousands of files in TEMP.  I don't; I prune them either manually or in the past (before W8 which I've not yet got all my normal scripts running in) automatically.  If an app starts creating files there, I will usually look at the files, and their time of creation, and see if I can understand why they are there.  Was something being installed for example?  Is such a file a log for some update being applied?  (If so I will rename it and keep it, if the thing being updated is significant).   

 

If you've turned the feature off, I thank-you.

[Pilis]

Isn't that what this Temp-directory (%USERPROFILE%\AppData\Local\Temp or %TEMP%) is for? A place where third-party programs can store their temporary files. Many many other programs temporarily store some files in there. There are all sorts of log-files getting created during installation or while updating a software.

I'm not sure what's the reason to "manage" those files other than regularly deleting the content of that folder (or letting Windows do that job for you).

 

Edit: You were faster, JeremyNicoll.

Edited August 16, 2015 by Pilis

[JeremyNicoll]

I keep log files (indeed in earlier versions of Windows I had registry keys set so that every MSI install would create a verbose log there even if not configured to do so), along with notes of what was being updated and when, because it's a useful resource for diagnosing problems.   'Managing' the files means deleting most of them, but keeping an eye on what gets created to see if some app has changed its behaviour, or perhaps some aspect of the OS has suddenly gone wrong or got turned on.   But if the owner/creator or files that get created in TEMP cannot readily be identified it's much harder to be properly aware of what's going on.   I wouldn't "let Windows" delete such files for me because it will not be intelligent about deciding which ones to delete.

[Fabian Wosar]

I have no problem with an app creating a temporary file, using it, and then deleting it.  But - for me - any app that creates files which hang around for long periods of time should ideally do so in a way that identifies what created it.

It is not like these files are hanging around. They are created on process start to swap some information that is exceptionally rarely used and that would otherwise be kept in RAM to disk. They are also created with the "Delete On Close" flag set. So the files are deleted by the file system driver automatically the moment the process that opened them either releases its file handle to the file or closes. So even if the process crashes because of an error before it could clean up, the file still ends up being deleted properly. The only way you can realistically end up with left over files from this is either a power outage or a blue screen. But in both these cases you likely have other, bigger problems.

[JeremyNicoll]

The files ARE hanging around; in TEMP now I see eleven of these files, all of which were created more than 8 hours ago.   Why aren't they in RAM taking advantage of virtual memory?

[Fabian Wosar]

The files ARE hanging around; in TEMP now I see eleven of these files, all of which were created more than 8 hours ago.

And when you close EAM they are gone? So how exactly is that "hanging around"? Do you object to the fact that they are there while they are used?

Why aren't they in RAM taking advantage of virtual memory?

I would assume you mean paging? Paging can be disabled by the user. You also, as an application developer, don't have full control over the paging algorithm that Windows uses. Maintaining your own file gives you more control, which usually results in much better performance as you can optimize your own paging strategy to your specific workflows and requirements.

[Peter2150]

Fabian, am I correct that these files are purged and recreated on reboot.   If so I just don't see it as a big deal.

[Fabian Wosar]

They are, yes.

[JeremyNicoll]

The thing is, I never close EAM, except when I shutdown the machine.  So what you're saying is that there will always be 11 weird-named files in temp...    As for managing priority in paging, doesn't the OS page-out little-used pages - surely it's meant to?   And you said that whatever's in the files is "some information that is exceptionally rarely used", but now you're saying you need to be able to control the files/pages for performance reasons.  Why does performance matter for little-used data?

[JeremyNicoll]

Anyway, it's not the existence of the files that annoys me so much as their random names.

[Fabian Wosar]

The thing is, I never close EAM, except when I shutdown the machine.  So what you're saying is that there will always be 11 weird-named files in temp...    As for managing priority in paging, doesn't the OS page-out little-used pages - surely it's meant to?   And you said that whatever's in the files is "some information that is exceptionally rarely used", but now you're saying you need to be able to control the files/pages for performance reasons.  Why does performance matter for little-used data?

Because they are rarely used, the conservative approach of the standard Windows paging mechanism is not the best choice. It tries to keep as much stuff in memory. We know however, that in 99% of all cases, we won't need that data during our process life time at all, so it really should be paged out immediately.

Anyway, it's not the existence of the files that annoys me so much as their random names.

Which is essentially personal preference. So unless we make the name of the temp files configurable there will always be someone who will be "annoyed" by the chosen file names.

[JeremyNicoll]

Aw, come off it.  "Personal preference" for app-specific names?  It's absolutely normal for OSes to use user/process/app-specific qualifiers in resource names.

[Fabian Wosar]

Aw, come off it.  "Personal preference" for app-specific names?

Well, you are the one that gets annoyed by randomly generated temporary file names. The same way you are annoyed by random file names, is it unfeasible that someone else is annoyed by whatever your preferred naming scheme is? In the end, I could understand being upset if we would completely clutter up the drive, because files aren't removed properly. But the way it is set up at the moment, files are removed properly. They just don't have a name you like. If that isn't personal preference, I don't know what is.

Bottom line is, even if we wanted to change it, we can't. It is unlikely that we will change error logging components because of your objections to the temporary file naming scheme either. We have suggested to the component provider to add a way to make at least part of the file name configurable. Whether he wants to spend resources on such a request is up to him and beyond our control. If he decides to implement it and if we can take advantage of such a feature without much effort on our part, we will consider it. In all other cases, we won't.

[JeremyNicoll]

The point is that a script reading the contents of TEMP to decide which files to delete cannot identify these files as EAM ones.  That's why I'd prefer a subdirectory name (or as someone else suggested an initial literal).  I don't get the impression that I'm the only person who thinks this way.

[Fabian Wosar]

A script couldn't remove those files anyways even if it wanted to, as they are locked and in use by the process that created them. If they weren't in use, they wouldn't exist anymore due to the fact that the Delete On Close flag was set when the file was originally created. If they exist, they are in use. If they aren't in use, they don't exist. Deletion is enforced by the operating system. The only way you can end up with orphaned temporary files are a power outage or a blue screen as mentioned before.

[JeremyNicoll]

I understand that "A script couldn't remove those files anyways", but the script is not able to identify which files it should not try to delete.   I don't generally program scripts so that they try things and rely on failure to bypass that attempt, but instead to identify items that shouldn't be processed.   I don't know if a failed attempt to delete something generate (say) an event record, but I'd be amazed if there wasn't some record in WMI or somewhere that a process tried to delete a locked file.  In some other OSes such an attempt would produce a session-disrupting error.

[Fabian Wosar]

I don't know if a failed attempt to delete something generate (say) an event record, but I'd be amazed if there wasn't some record in WMI or somewhere that a process tried to delete a locked file.

It is possible to audit file system events, but it has to be enabled beforehand. It is not enabled by default. You could actually use that very mechanism to track which process created which file to feed your cleanup script. That would surely beat trying to assign them by name.

If you don't want to do that for your script, the best approximation to detect files is to match the file name via regex. ^[0123456789ABCDEF]{32}\.tmp$ should do the trick just fine. It is not perfect. But then again, neither would be "tagging" files somehow. We are not the only one using the a2 prefix.

 

In some other OSes such an attempt would produce a session-disrupting error.

In some other OSes the temporary files folder would be stored in shared memory and we wouldn't have this conversation.

[Peter2150]

Jeremy, there is an old texas adage.   No matter how much you kick a dead horse it  ain't getting up.

[iammike]

Thanks for reverting to an older version, but I for one (and I am not alone) would like to see those files created in a Subdir (like a2temp) which you already do now with the sigs.

 

The reason why I want this, is not that I want my TEMP look nice and tidy, but it was a shock to me what those files where ?

 

Did I got a virus ? etc etc etc. Fortunately I know how to identify those locked files, but many users will not be able to and thus this could lead to more support calls.

 

But as Peter2150 said, "No matter how much you kick a dead horse it  ain't getting up."

[JeremyNicoll]

Auditing file system events sounds good (computing-wise much of my professional life was spent in large IBM mainframe sites where this sort of logging happened as a matter of course, making it possible to determine who damaged a file weeks or months after the event)...  Of course the volume of data collected in those logs was vast, but it was also very useful, for those allowed to read it.

 

I'm not sure I'd want to try to read W8's records in real time, for the %TEMP% monitoring script, but it's certainly something I'll pursue for general monitoring of what's going on in the system.  Like

so much else in Windows, finding out what's possible is half the battle.

 

 

[under XP, where I had 'Pro', I was able to turn on auditing of 'Security' events which included user login, process creation & termination.  That's really useful for seeing scripts, scheduled tasks etc starting & stopping especially as the Security eventlog record contained the full commandline and - along with debug logs created by my own apps that recorded the process-ids they were running under, one could track down what happened when on a system much more easily. 

 

My W8.1 system however, only has W8.1 Home on it, and although the OS support for creating audit records is present in W8.1 Home, the tool normally used to turn these things on and off - gpedit.msc - isn't available / supported for W8.1 Home (Google suggests that it might work, if you're lucky, if you copy the relevant files from a Pro/Ultimate/Enterprise system, if you can get access to one.

 

 

However I found a while ago that the command-line tool: auditpol.exe, run from an elevated command prompt DOES allow you to query and alter the settings for this sort of thing.  So for example:

 

Auditpol.exe /set /subcategory:"process creation" /success:enable
Auditpol.exe /set /subcategory:"process creation" /failure:enable

- are the commands I used here to turn on generation of eventlog records every time an OS process creation either works or fails; and


Auditpol.exe /set /subcategory:"process termination" /success:enable
Auditpol.exe /set /subcategory:"process termination" /failure:enable

- turns on generation of eventlog records for successful/failed process termination.

 

 

One can also see the whole set of what's enabled and disabled by, eg:

 

C:\Windows\system32>auditpol /get /category:*
System audit policy
Category/Subcategory                      Setting
System
  Security System Extension               No Auditing
  System Integrity                        Success and Failure
  IPsec Driver                            No Auditing
  Other System Events                     Success and Failure
  Security State Change                   Success
Logon/Logoff
  Logon                                   Success
  Logoff                                  Success
  Account Lockout                         Success
  IPsec Main Mode                         No Auditing
  IPsec Quick Mode                        No Auditing
  IPsec Extended Mode                     No Auditing
  Special Logon                           Success
  Other Logon/Logoff Events               No Auditing
  Network Policy Server                   Success and Failure
  User / Device Claims                    No Auditing
Object Access
  File System                             No Auditing
  Registry                                No Auditing
  Kernel Object                           No Auditing
  SAM                                     No Auditing
  Certification Services                  No Auditing
  Application Generated                   No Auditing
  Handle Manipulation                     No Auditing
  File Share                              No Auditing
  Filtering Platform Packet Drop          No Auditing
  Filtering Platform Connection           No Auditing
  Other Object Access Events              No Auditing
  Detailed File Share                     No Auditing
  Removable Storage                       No Auditing
  Central Policy Staging                  No Auditing
Privilege Use
  Non Sensitive Privilege Use             No Auditing
  Other Privilege Use Events              No Auditing
  Sensitive Privilege Use                 No Auditing
Detailed Tracking
  Process Creation                        Success and Failure
  Process Termination                     Success and Failure
  DPAPI Activity                          No Auditing
  RPC Events                              No Auditing
Policy Change
  Authentication Policy Change            Success
  Authorization Policy Change             No Auditing
  MPSSVC Rule-Level Policy Change         No Auditing
  Filtering Platform Policy Change        No Auditing
  Other Policy Change Events              No Auditing
  Audit Policy Change                     Success
Account Management
  User Account Management                 Success
  Computer Account Management             No Auditing
  Security Group Management               Success
  Distribution Group Management           No Auditing
  Application Group Management            No Auditing
  Other Account Management Events         No Auditing
DS Access
  Directory Service Changes               No Auditing
  Directory Service Replication           No Auditing
  Detailed Directory Service Replication  No Auditing
  Directory Service Access                No Auditing
Account Logon
  Kerberos Service Ticket Operations      No Auditing
  Other Account Logon Events              No Auditing
  Kerberos Authentication Service         No Auditing
  Credential Validation                   No Auditing

C:\Windows\system32>

 

 

 

To get the full commandline logged when a process starts, I used the (excellent) tool at: http://gpsearch.azurewebsites.net/#10674

to find out how command-line logging is controlled.  It says

  System
     Auditing
        Include command line in process creation events

        HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\Audit

        Value: ProcessCreationIncludeCmdLine_Enabled

but this doesn't say what sort of value that should have.     I googled and found that  a  DWORD with value 1 is required.   After

creating that registry entry, my W8.1 Security eventlog records for process creation now contain the command lines.  Anyone

who runs complicated scripts that issue commands that run other scripts with varying commandlines will benefit from having all

of that logged!

 

 

Anyway, I expect that googling will tell me how to make use of file system auditing; I expect it can be made fairly granular, monitoring

either all files & all users, or specific subsets.

[iammike]

This morning I got version 10.0.0.5641 and the temp files are gone (for how long we don't know )

[Fabian Wosar]

Until the first stable update that uses the new version again.

[iammike]

Until the first stable update that uses the new version again.

 

Thanks, Fabian.

 

No reconsideration on this, only that it will be coming back ?

[Peter2150]

Considering how many temp files that are created that don't go away, why is this such a big deal?

[Fabian Wosar]

No reconsideration on this, only that it will be coming back ?

I made myself pretty clear: Even if we wanted to change it, we can't. It is unlikely that we will change error logging components because of your objections to the temporary file naming scheme either. We have suggested to the component provider to add a way to make at least part of the file name configurable. Whether he wants to spend resources on such a request is up to him and beyond our control. If he decides to implement it and if we can take advantage of such a feature without much effort on our part, we will consider it. In all other cases, we won't.

[iammike]

Oke understood.

[Fabian Wosar]

Great .