Date: Fri, 8 Sep 95 12:39 PDT X-PH: V4.1@cornell.edu (Cornell Modified) From: markm@cse.ogi.edu (Mark Morrissey) To: fors-discuss@teleport.com, jeffrey@best.com, sp17@cornell.edu (Steven Pacenka) Subject: Intel report on Schwartz I was asked to post to the FORS list this report that I wrote while at Intel. The reason for making this report public is that it specifically mentions that Randal was using Intel resources to crack password files from at least one other company. In addition to cracking O'Reilly, Randal has admitted to cracking Teleport. I have no information as to where Randal ran crack on the Teleport system, but he did use Intel resources to crack O'Reilly. I think that it is fairly obvious from this report that I didn't see any clear indication of a violation of the law. However, if one reads the search warrant (that I didn't see until after it was served), there is a statement attributed to me where I supposedly tell law enforcement that Randal has violated Oregon law. I never made such a statement to law enforcement. ----------------------------------------------------------------------- Report on a Security Incident at the Oregon Facility Mark Morrissey November 3, 1993 SUMMARY On Thursday, October 28, at 12:30 in the afternoon, I noticed an unusual process running on a Sun computer which I administer. Further checking convinced me that this was a program designed to break, or crack, passwords. I was able to determine that the user "merlyn" was running the program. The username "merlyn" is assigned to Randal Schwartz, an independent contractor. The password cracking program had been running since October 21st. I investigated the directory from which the program was running and found the program to be Crack 4.1, a powerful password cracking program. There were many files located there, including passwd.ssd and passwd.ora. Based on my knowledge of the user, I guessed that these were password files for the Intel SSD organization and also an external company called O'Reilly and Associates. I then contacted Rich Cower in Intel security. By 1:30 I was in contact with John Kent at SSD. John confirmed that Randal did not have permission to crack password files from SSD. I told John that I had noticed logins to my machines by Randal over a period of months from an SSD machine called "brillig." John confirmed that Randal did not have permission for this activity. I asked John to check for a backdoor program called "gate" which would allow Randal to gain access to Intel computers from outside of Intel. John did find this program as well as log files which showed connections to Intel from a machine called ruby.ora.com (operated by O'Reilly and Associates). At this time we decided that two security violations existed. Historically, the user name for Randal Schwartz at Intel is merlyn. The processes running on brillig that allowed unauthorized access from outside of the Intel network were owned by merlyn. The actual programs were located in the merlyn directory. On my systems, the crack program was found in the merlyn directory. The crack process found running on my system was owned by merlyn. John Kent and I feel that we can demonstrate two issues: 1. Randal Schwartz has, over a period of time, gained access to Intel computer systems from at least one computer system outside of Intel in violation of Intel policy. 2. Randal Schwartz did crack a password file from the SSD organization and possibly an outside company. In the case of SSD, at least 40 passwords were compromised. For the outside company, at least one password was compromised. We cannot show that Randal has acted using the cracked passwords. Similarly, we cannot show that he has not made use of them. On Monday, November 1st, I cooperated with Washington County and Intel authorities to prepare an affidavit to be used for securing a search warrant. This search warrant was executed that evening. On Tuesday, November 2nd, Rich Cower informed me that Randal Schwartz did admit to cracking SSD passwords. Rich told me that Randal Schwartz also admitted to cracking passwords for two external companies. I continue to cooperate with Intel and appropriate external entities in the investigation of this incident. DETAIL On Thursday, October 28, at 12:30 in the afternoon, I logged on to a machine called "snoopy." This machine is a recently installed Sun SparcStation 10 model 51 running the UNIX operating system. This machine was purchased to run the server portion of Cabletron's network management application, Spectrum. Snoopy has been operating since October 14. Part of my responsibilities in the Oregon SIT/NTU organization is systems administration for NTU UNIX systems at Hawthorn Farms. On Thursday, I had two reasons for logging on to snoopy: 1) to ensure that the Spectrum server was operating correctly and that no further system modifications were required; and 2) to make sure that Randal Schwartz had not moved any of his programs to this machine as Randal has a habit of using as much CPU power as he can find. Randal had been previously asked not to run jobs which could interfere with the Spectrum server once snoopy had been installed. I executed a command to list the processes on snoopy. I was not surprised to find a process owned by merlyn. This process had been running since October 21st. I executed another command which would allow me to see what command merlyn was running. To my surprise, the running command was called crack-pwc. Given that there is a UNIX password cracking program called crack, I became suspicious and decided that I should investigate. The program was executing from the directory /two/usrmerlyn/play/cr/sparc. I went to that location and discovered that the cr directory contained the newest version of the crack program. I also discovered two suspicious files: passwd.ssd and passwd.ora. These appeared to be UNIX password files. I know that Randal has previously contracted at SSD and that he has an account on a system owned by his book publisher, O'Reilly and Associates. O'Reilly goes by ORA and has the email address of ora.com. At this time, I contacted Rich Cower in corporate security to receive instructions on how to proceed. Rich and I decided that this was a serious problem. Rich suggested that I contact Lou Poehlitz or John Kent at SSD to inform then of what I knew. I contacted Lou, who directed me to John Kent. John confirmed that Randal should not be in possession of SSD password files and did not have permission from John to crack passwords. John mentioned that running the crack program is, with only limited exceptions, a firing offense at SSD. While talking to John, I mentioned that I had seen several logins by Randal from an SSD machine called brillig. John was alarmed and stated that all of Randal's accounts should have been removed after his contract expired the previous spring. John also mentioned that Randal received a severe reprimand within a week of his contract expiring for a security incident at SSD. I instructed John to look for a program called gate running on brillig. This past spring, Randal was found to be running this program on a machine at ADL which has Internet access. The program can be used to gain access to the Intel network from computers outside of Intel. The use of this program on the ADL machine eventually resulted in the removal of Randal's account from the ADL machine. See the end of this document for a summary of that incident. John did find the gate program running on brillig and also found log files indicating that Randal had used brillig to gain access to the Intel network on many occasions from a machine called ruby.ora.com. This machine is operated by O'Reilly and Associates, a publisher of UNIX books. Seeing that access from an external company was occurring and suspecting that Randal was cracking the O'Reilly password file, I asked Rich Cower to contact CERT to ask for advice and to inform them that we were tracking a potential security threat to O'Reilly as well as Intel and that unauthorized access to the Intel network had been achieved from an O'Reilly machine. Rich later informed me that CERT would make contact, but that Intel's name would not be used. I was asked to be prepared to provide information related to O'Reilly to CERT when requested. I decided to inform Oregon IT management of this incident. I contacted Bob Wilcox (Randal's manager), John Gray (HF campus IT owner), and Mike Moon (Oregon Site IT owner). Rick Query (Oregon SIT/NTU) was present when I talked with Mike. Much later on Thursday, I also informed Brad Benson (SIT/SAU owner, my manager) that I was investigating a security incident. I made clear to everyone that I felt that the security organization needed to run the show. I insisted that Rich Cower direct the activities for the short term. On Friday morning, I informed Merlon Altermatt and Bill Morgan that backup tapes used to backup my machines were no longer to be reused. This step was to ensure that I had daily backups of Randal's directories going back as far as possible. I contacted HR legal to ensure that we weren't doing anything which was either illegal or against Intel policy. Coeta Chambers concurred that we were operating correctly. By 1:00 on Friday afternoon, I had given the information pertaining to O'Reilly and Associates to CERT. No Intel information was given to CERT. At 3:30 on Friday, a bridge meeting was held to discuss the situation. The activities were shown to be serious. The group opted not to make any changes over the weekend which were likely to be discovered by Randal should he log on to our systems or if he had a watchdog program installed. The decision was made to have everyone involved ready to move on Monday, November 1, if that proved necessary. Saturday afternoon I logged on to snoopy to check the progress of the crack program. The program was not running. My calculations, based on the log file for crack, showed that the program should have been running for several more weeks. I admit that I do not know enough to determine if the program terminated normally, abnormally, or was stopped by Randal or others. The fact that the program terminated in the middle of an investigation into the program was unsettling to me. I left messages for John Kent and Rich Cower and asked that brillig be checked for activity. On Monday, November 1st, I met with Rich Cower, Rick Pierce, Clyde Stites, and John Kent to discuss the situation and bring everyone up-to-date. Washington County authorities were briefed later in the morning and onsite before the afternoon. I cooperated with the Washington County authorities in writing an affidavit which was to be used to secure a search warrant. I was informed that there was a very high probability that the search warrant would be executed Monday evening. I physically shut down the six computers which I control at 5:30 on Monday evening. At 6:30 pm on Monday, I was informed that the search warrant had been executed. On Tuesday, November 2nd, I met with Rich Cower and Clyde Stites to discuss how to ensure that my systems were secure and also to make sure that I maintained all information which might be of use to Washington County authorities. During this meeting, Washington County authorities arrived to present a search warrant. By Tuesday afternoon, I had scanned all system files to make sure that no backdoors had been installed. At that time, I brough my systems back online, changed all password, disabled the merlyn login and secured all locations where files relating to this incident were stored. I secured the locations using the UNIX "chmod" command and setting permissions to "000" which allows no access. ADL Security Incident About March of this year, Dirk Brandewie from ADL noticed a long running process on a machine called mink, which Dirk administers. Dirk's investigation showed that this program was accepting connections from outside of Intel. The process and program were owned by Randal Schwartz. Dirk and Mark Morrissey confronted Randal, who agreed to add code which would ensure that only connections from within Intel would be accepted. Dirk followed up to ensure that the changes were made. Rich Cower was advised at that time that a security threat had been found and dealt with. In the July time frame, Dirk rechecked the program and found the security checks removed. Dirk confronted Randal a second time. Randal explained that the program was being used to accept X Window connections from an O'Reilly and Associates machine named ruby.ora.com. Dirk informed Randal that connections from outside of Intel would not be allowed. Randal requested that his account on mink be removed as outside access was the only reason for having that account. CONCLUSIONS We can demonstrate that Randal Schwartz has been gaining access to Intel systems via a mechanism he has previously been informed is unacceptable. The access mechanism on brillig is identical to the one used on mink. We do not know at this time if other backdoors have been installed elsewhere on Intel machines. We can demonstrate that Randal has run a password cracking program against SSD, and possibly ORA, password files. For the SSD password file, we can show that he did not have permission to do so. The act of cracking password files can have two motives: 1) enhancing local security by identifying insecure passwords and encouraging users to change them to be more secure; and 2) a desire to find out passwords. Cracking password files without explicit direction or permission from appropriate sources can be interpreted as a hostile act. I do not know if Randal has permission to crack O'Reilly passwords. He does not have permission from either his management or myself to crack Intel passwords. Similarly, he does not have permission to use Intel computing resources to crack passwords on behalf of any external entity. I have no evidence at this time that Randal has acted in concert or with the cooperation of others. Similarly, I have no evidence that he has acted on his own. I cannot determine if any other password files have been cracked by Randal.