diff --git a/trackingPaper.tex b/trackingPaper.tex index 028ebcc..32c2b64 100644 --- a/trackingPaper.tex +++ b/trackingPaper.tex @@ -131,7 +131,7 @@ University of Connecticut, USA\\ \begin{abstract} Storage system traces are important for examining real-world applications, studying potential bottlenecks, as well as driving benchmarks in the evaluation of new system designs. While file system traces have been well-studied in earlier work, it has been some time since the last examination of the SMB network file system. -The purpose of this work is to continue previous SMB studies to better understand the use of the protocol in a real-world production system in use at a major research university. %\textcolor{red}{the University of Connecticut}. +The purpose of this work is to continue previous SMB studies to better understand the use of the protocol in a real-world production system in use at the University of Connecticut. The main contribution of our work is the exploration of I/O behavior in modern file system workloads as well as new examinations of the inter-arrival times and run times for I/O events. We further investigate if the recent standard models for traffic remain accurate. Our findings reveal interesting data relating to the number of read and write events. We notice that the number of read and write events is significantly less than creates and the average number of bytes exchanged per I/O is much smaller than what has been seen in previous studies. @@ -167,8 +167,8 @@ Since an SMB-based trace study has not been undertaken recently, we took a look at its current implementation and use in a large university network. %Due to the sensitivity of the captured information, we ensure that all sensitive information is hashed and that the original network captures are not saved. -Our study is based on network packet traces collected on a major research university's -%\textcolor{red}{the University of Connecticut}'s +Our study is based on network packet traces collected on +the University of Connecticut's centralized storage facility over a period of three weeks in May 2019. This trace-driven analysis can help in the design of future storage products as well as providing data for future performance benchmarks. %Benchmarks are important for the purpose of developing technologies as well as taking accurate metrics. The reasoning behind this tracing capture work is to eventually better develop accurate benchmarks for network protocol evaluation. Benchmarks allow for the stress testing of various aspects of a system (e.g. network, single system). Aggregate data analysis collected from traces can lead to the development of synthetic benchmarks. Traces can also expose systems patterns that can also be reflected in synthetic benchmarks. Finally, the traces themselves can drive system simulations that can be used to evaluate prospective storage architectures. @@ -191,11 +191,11 @@ Benchmarks allow for the stress testing of various aspects of a system (e.g. net We created a new tracing system to collect data from the university %\textcolor{red}{UConn} -storage network system. The tracing system was built around the high-speed PF\_RING packet capture system and required the use of proper hardware and software to handle incoming data%\textcolor{blue}{; however interaction with later third-party code did require re-design for processing of the information} +storage network system. The tracing system was built around the high-speed PF\_RING packet capture system~\cite{pfringWebsite} and required the use of proper hardware and software to handle incoming data%\textcolor{blue}{; however interaction with later third-party code did require re-design for processing of the information} . We also created a new trace capture format based on the DataSeries structured data format developed by HP~\cite{DataSeries}. % PF\_RING section %The addition of PF\_RING lends to the tracing system by minimizing the copying of packets which, in turn, allows for more accurate timestamping of incoming traffic packets being captured ~\cite{Orosz2013,skopko2012loss,pfringWebsite,PFRINGMan}. -PF\_RING acts as a kernel module that aids in minimizing packet loss/timestamping issues by not passing packets through the kernel data structures~\cite{PFRINGMan}. +PF\_RING acts as a kernel module that aids in minimizing packet loss/timestamping issues by not passing packets through the kernel data structures. %The other reason PF\_RING is instrumental is that it functions with the 10Gb/s hardware that was installed into the Trace1 server; allowing for full throughput from the network tap on the UITS system. \\ % DataSeries + Code section DataSeries was modified to filter specific SMB protocol fields along with the writing of analysis tools to parse and dissect the captured packets. Specific fields were chosen to be the interesting fields kept for analysis. @@ -239,9 +239,9 @@ This paper & 2020 & SMB & x & Dynamic & We summarize major works in trace study in Table~\ref{tbl:studySummary}. %In addition we examine issues that occur with traces and the assumptions in their study. Tracing collection and analysis from previous studies have provided important insights and lessons such as an observations of read/write event changes, overhead concerns originating in system implementation, bottlenecks in communication, and other revelations found in the traces. -Previous tracing work has shown that one of the largest and broadest hurdles to tackle is that traces (and benchmarks) must be tailored to the system being tested. There are always some generalizations taken into account but these generalizations can also be a major source of error (e.g. timing, accuracy, resource usage) ~\cite{vogels1999file,malkani2003passive,seltzer2003nfs,anderson2004buttress,Orosz2013,dabir2007bottleneck,skopko2012loss,traeger2008nine,ruemmler1992unix}. +Previous tracing work has shown that one of the largest and broadest hurdles to tackle is that traces (and benchmarks) must be tailored to the system being tested. There are always some generalizations taken into account, but these generalizations can also be a major source of error (e.g. timing, accuracy, resource usage) ~\cite{vogels1999file,malkani2003passive,seltzer2003nfs,anderson2004buttress,Orosz2013,dabir2007bottleneck,skopko2012loss,traeger2008nine,ruemmler1992unix}. To produce a benchmark with high fidelity one needs to understand not only the technology being used but how it is being implemented within the system~\cite{roselli2000comparison,traeger2008nine,ruemmler1992unix}. All these aspects lend to the behavior of the system; from timing and resource elements to how the managing software governs actions~\cite{douceur1999large,malkani2003passive,seltzer2003nfs}. Furthermore, in pursuing this work one may find unexpected results and learn new things through examination~\cite{leung2008measurement,roselli2000comparison,seltzer2003nfs}. -These studies are required in order to evaluate the development of technologies and methodologies along with furthering knowledge of different system aspects and capabilities. As has been pointed out by past work, the design of systems is usually guided by an understanding of the file system workloads and user behavior~\cite{leung2008measurement}. +These studies are required in order to evaluate the development of technologies and methodologies along with furthering knowledge of different system aspects and capabilities. As has been pointed out by past work, the design of systems is usually guided by an understanding of the file system workloads and user behavior. %It is for that reason that new studies are constantly performed by the science community, from large scale studies to individual protocol studies~\cite{leung2008measurement,vogels1999file,roselli2000comparison,seltzer2003nfs,anderson2004buttress}. Even within these studies, the information gleaned is only as meaningful as the considerations of how the data is handled. %The work done by @@ -249,7 +249,7 @@ Leung et al.~\cite{leung2008measurement} found that %observations related to the infrequency of files to be shared by more than one client. over 67\% of files were never opened by more than one client. %Work by Leung \textit{et al.} led to a series of observations, from the fact that files are rarely re-opened to finding -and that read-write access patterns are more frequent ~\cite{leung2008measurement}. +and that read-write access patterns are more frequent. %If files were shared it was rarely concurrently and usually as read-only; where 5\% of files were opened by multiple clients concurrently and 90\% of the file sharing was read only. %Concerns of the accuracy achieved of the trace data was due to using standard system calls as well as errors in issuing I/Os leading to substantial I/O statistical errors. % Anderson Paper @@ -262,7 +262,7 @@ Issues of inaccuracies in scheduling I/O can result in as much as a factor 3.5 d Orosz and Skopko examined the effect of the kernel on packet loss and %in their 2013 paper~\cite{Orosz2013}. Their work -showed that when taking network measurements the precision of the timestamping of packets is a more important criterion than low clock offset, especially when measuring packet inter-arrival times and round-trip delays at a single point of the network. One solution for network capture is the tool Dumpcap. However the concern with Dumpcap is that it is a single threaded application and was suspected to be unable to handle new arriving packets due to the small size of the kernel buffer. Work by +showed that when taking network measurements, the precision of the timestamping of packets is a more important criterion than low clock offset, especially when measuring packet inter-arrival times and round-trip delays at a single point of the network. One solution for network capture is the tool Dumpcap. However, the concern with Dumpcap is that it is a single threaded application and was suspected to be unable to handle new arriving packets due to the small size of the kernel buffer. Work by Dabir and Matrawy%, in 2008 ~\cite{dabir2007bottleneck} attempted to overcome this limitation by using two semaphores to buffer incoming strings and improve the writing of packet information to disk. %Narayan and Chandy examined the concerns of distributed I/O and the different models of parallel application I/O. @@ -271,11 +271,12 @@ Dabir and Matrawy%, in 2008 %Observations from Skopk\'o %in a 2012 paper -~\cite{skopko2012loss} examined the concerns of software based capture solutions and observed that +~\cite{skopko2012loss} examined the concerns of software-based capture solutions and observed that %. The main observation was software solutions relied heavily on OS packet processing mechanisms. Furthermore, depending on the mode of operation (e.g. interrupt or polling), the timestamping of packets would change. -As seen in previous trace work~\cite{leung2008measurement,roselli2000comparison,seltzer2003nfs}, the general perceptions of how computer systems are being used versus their initial purpose have allowed for great strides in eliminating actual bottlenecks rather than spending unnecessary time working on imagined bottlenecks. Without illumination of these underlying actions (e.g. read-write ratios, file death rates, file access rates) these issues can not be readily tackled. +As seen in previous trace work~\cite{leung2008measurement,roselli2000comparison,seltzer2003nfs}, the general perceptions of how computer systems are being used versus their initial purpose have allowed for great strides in eliminating actual bottlenecks rather than spending unnecessary time working on imagined bottlenecks. Without illumination of these underlying actions (e.g. read-write ratios, file death rates, file access rates) these +issues cannot be readily tackled. \section{Background} @@ -335,8 +336,8 @@ Some nuances of the SMB protocol I/O to note are that SMB/SMB2 write requests ar \subsection{University Storage System Overview} -We collected traces from the university -%\textcolor{red}{the University of Connecticut University Information Technology Services (UITS)} +We collected traces from +the University of Connecticut Information Technology Services (ITS) centralized storage server%The \textcolor{red}{UITS system} , which consists of five Microsoft file server cluster nodes. These blade servers are used to host SMB file shares for various departments at the university @@ -362,7 +363,7 @@ We had to tune an implementation of \texttt{tshark} (wireshark's terminal pcap i \texttt{tshark} outputs \texttt{.pcap} files which captures all of the data present in packets on the network. We configure \texttt{tshark} so that it only captures SMB packets. Furthermore, to optimize this step, a capture ring buffer flag is used to minimize the amount of space used to write \texttt{.pcap} files, while optimizing the amount of time to %\textit{pcap2ds} can filter data from the \texttt{.pcap} files. -The filesize used was in a ring buffer where each file captured was 64000 kB. +The file size used was in a ring buffer where each file captured was 64000 kB. % This causes tshark to switch to the next file after it reaches a determined size. %To simplify this aspect of the capturing process, the entirety of the capturing, dissection, and permanent storage was all automated through watch-dog scripts. @@ -420,7 +421,7 @@ Table~\ref{tbl:TraceSummaryTotal} show a summary of the SMB traffic captured, statistics of the I/O operations, and read/write data exchange observed for the network filesystem. This information is further detailed in Table~\ref{tbl:SMBCommands}, which illustrates that the majority of I/O operations are general (74.87\%). As shown in %the bottom part of Table~\ref{tbl:SMBCommands2}, general I/O includes metadata commands such as connect, close, query info, etc. -Our examination of the collected network filesystem data revealed interesting patterns for the current use of CIFS/SMB in a large engineering academic setting. The first is that there is a major shift away from read and write operations towards more metadata-based ones. This matches the last CIFS observations made by Leung et.~al.~that files were being generated and accessed infrequently. The change in operations are due to a movement of use activity from reading and writing data to simply checking file and directory metadata. However, since the earlier study, SMB has transitioned to the SMB2 protocol which was supposed to be less "chatty" and thus we would expect fewer general SMB operations. Table~\ref{tbl:SMBCommands} shows a breakdown of SMB and SMB2 usage over the time period of May. From this table, one can see that the SMB2 protocol makes up $99.14$\% of total network operations compared to just $0.86$\% for SMB, indicating that most clients have upgraded to SMB2. However, $74.66$\% of SMB2 I/O are still general operations. Contrary to the purpose of implementing the SMB2 protocol, there is still a large amount of general I/O. +Our examination of the collected network filesystem data revealed interesting patterns for the current use of CIFS/SMB in a large academic setting. The first is that there is a major shift away from read and write operations towards more metadata-based ones. This matches the last CIFS observations made by Leung et.~al.~that files were being generated and accessed infrequently. The change in operations are due to a movement of use activity from reading and writing data to simply checking file and directory metadata. However, since the earlier study, SMB has transitioned to the SMB2 protocol which was supposed to be less "chatty". As a result, we would expect fewer general SMB operations. Table~\ref{tbl:SMBCommands} shows a breakdown of SMB and SMB2 usage over the time period of May. From this table, one can see that the SMB2 protocol makes up $99.14$\% of total network operations compared to just $0.86$\% for SMB, indicating that most clients have upgraded to SMB2. However, $74.66$\% of SMB2 I/O are still general operations. Contrary to the purpose of implementing the SMB2 protocol, there is still a large amount of general I/O. %While CIFS/SMB protocol has less metadata operations, this is due to a depreciation of the SMB protocol commands, therefore we would expect to see less total operations (e.g. $0.04$\% of total operations). %The infrequency of file activity is further strengthened by our finding that within a week long window of time there are no Read or Write inter arrival times that can be calculated. %\textcolor{red}{XXX we are going to get questioned on this. its not likely that there are no IATs for reads and writes} @@ -493,7 +494,9 @@ Cancel & \multicolumn{2}{|c|}{0} & 0.00\% \\ %\end{figure} Each SMB Read and Write command is associated with a data request size that indicates how many bytes are to be read or written as part of that command. Figure~\ref{fig:SMB-Bytes-IO} %and~\ref{fig:PDF-Bytes-Write} -shows the probability density function (PDF) of the different sizes of bytes transferred for read and write I/O operations respectively. The most noticeable aspect of these graphs are that the majority of bytes transferred for read and write operations is around 64 bytes. It is worth noting that write I/Os also have a larger number of very small transfer amounts. This is unexpected in terms of the amount of data passed in a frame. Part of the reason is due to a large number of long term %calculations/ +shows the probability density function (PDF) of the different sizes of bytes transferred for read and write I/O operations respectively. The most noticeable aspect of these +graphs is that the majority of bytes transferred for read and write operations is around 64 bytes. It is worth noting that write I/Os also have a larger number of very small transfer amounts. This is unexpected in terms of the amount of data passed in a frame. +Part of the reason is due to a large number of long-term %calculations/ scripts that only require small but frequent updates, as we observed several %. This assumption was later validated in part when examining the files transferred, as some were related to running scripts creating a large volume of files. A more significant reason was because we noticed Microsoft Word would perform a large number of small reads at ever growing offsets. This was interpreted as when a user is viewing a document over the network and Word would load the next few lines of text as the user scrolled down the document; causing ``loading times'' amid use. Finally, a large degree of small writes were observed to be related to application cookies or other such smaller data communications. @@ -555,11 +558,11 @@ running scripts creating a large volume of files. A more significant reason was % \label{fig:CDF-Bytes-RW} %\end{figure} Figure~\ref{fig:SMB-Bytes-IO} %and~\ref{fig:CDF-Bytes-Write} -shows cumulative distribution functions (CDF) for bytes read and bytes written. As can be seen, almost no read transfer sizes are less than 32 bytes, whereas 20\% of the writes are smaller than 32 bytes. Table~\ref{fig:transferSizes} shows a tabular view of this data. For reads, $34.97$\% are between 64 and 512 bytes, with another $28.86$\% at 64 byte request sizes. There are a negligible percentage of read requests larger than 512. +shows cumulative distribution functions (CDF) for bytes read and bytes written. As can be seen, almost no read transfer sizes are less than 32 bytes, whereas 20\% of the writes are smaller than 32 bytes. Table~\ref{fig:transferSizes} shows a tabular view of this data. For reads, $34.97$\% are between 64 and 512 bytes, with another $28.86$\% at 64-byte request sizes. There are a negligible percentage of read requests larger than 512. This read data differs from the size of reads observed by Leung et al. by a factor of four smaller. %This read data is similar to what was observed by Leung et al, however at an order of magnitude smaller. %Writes observed also differ from previous inspection of the protocol's usage. % are very different. -Leung et al. showed that $60$-$70$\% of writes were less than 4K in size and $90$\% less than 64K in size. In our data, however, we see that almost all writes are less than 1K in size. In fact, $11.16$\% of writes are less than 4 bytes, $52.41$\% are 64 byte requests, and $43.63$\% of requests are less than 64 bytes. +Leung et al. showed that $60$-$70$\% of writes were less than 4K in size and $90$\% less than 64K in size. In our data, however, we see that almost all writes are less than 1K in size. In fact, $11.16$\% of writes are less than 4 bytes, $52.41$\% are 64-byte requests, and $43.63$\% of requests are less than 64 bytes. In the ten years since the last study, it is clear that writes have become significantly smaller. In our analysis of a subset of the writes, we found that a significant part of the write profile was writes to cookies which are necessarily small files. The preponderance of web applications and the associated tracking is a major change in how computers and data storage are used compared to a decade ago. These small data reads and writes significantly alter the assumptions that most network storage systems are designed for. %This may be explained by the fact that large files, and multiple files, are being written as standardized blocks more fitting to the frequent update of larger data-sets and disk space available. This could be as an effort to improve the fidelity of data across the network, allow for better realtime data consistency between client and backup locations, or could just be due to a large number of scripts being run that create and update a series of relatively smaller documents. %\textbf{Note: It seems like a change in the order of magnitude that is being passed per packet. What would this indicate?}\textcolor{red}{Answer the question. Shorter reads/writes = better?} @@ -584,7 +587,7 @@ $= 1024$ & 1.22e-5\% & 3.81e-5\% \\ \hline In comparison of the read, write, and create operations we found that the vast majority of these type of I/O belong to creates. By the fact that there are so many creates, it seems apparent that many applications create new files rather than updating existing -files when files are modified. Furthermore, read operations account for the largest aggregate of bytes transferred over the network. However, the amount of bytes transferred by write commands is not far behind, although, non-intuitively, including a larger number of standardized relatively smaller writes. The most unexpected finding of the data is that all the the read and writes are performed using much smaller buffers than expected; about an order of magnitude smaller (e.g. bytes instead of kilobytes). +files when files are modified. Furthermore, read operations account for the largest aggregate of bytes transferred over the network. However, the number of bytes transferred by write commands is not far behind, although, non-intuitively, including a larger number of standardized relatively smaller writes. The most unexpected finding of the data is that all the read and writes are performed using much smaller buffers than expected; about an order of magnitude smaller (e.g. bytes instead of kilobytes). % XXX I think we should get rid of this figure - not sure it conveys anything important that is not better conveyed than the CDF %Figure~\ref{fig:Agg-AvgRT} shows the average response time (RT) for the different I/O operations. The revealing information is that write I/Os take the longest average time. This is expected since writes transfer more data on average. There is an odd spike for create I/O which can be due to a batch of files or nested directories being made. There are points where read I/O RT can be seen, but this only occurs in areas where large RT for write I/O occur. This is attributed to a need to verify the written data. @@ -620,7 +623,7 @@ files when files are modified. Furthermore, read operations account for the lar \subsection{I/O Response Times} %~!~ Addition since Chandy writing ~!~% -Most previous tracing work has not reported I/O response times or command latency which is generally proportional to data request size, but under load, the response times give an indication of server load. In +Most previous tracing work has not reported I/O response times or command latency, which is generally proportional to data request size, but under load, the response times give an indication of server load. In Table~\ref{tbl:PercentageTraceSummary} we show a summary of the response times for read, write, create, and general commands. We note that most general (metadata) operations occur fairly frequently, run relatively slowly, and happen at high frequency. We also observe that the number of writes is very close to the number of reads. The write response time for their operations is very small - most likely because the storage server caches the write without actually committing to disk. Reads, on the other hand, are in most cases probably not going to hit in the cache and require an actual read from the storage media. Although read operations are only a small percentage of all operations, they have the highest average response time. As noted above, creates happen more frequently, but have a slightly slower response time, because of the extra metadata operations required for a create as opposed to a simple write. @@ -912,7 +915,7 @@ However, one should notice that the response time on read operations grows at a \subsection{File Extensions} Tables~\ref{tab:top10SMB2FileExts} and~\ref{tab:commonSMB2FileExts} show a summary of the various file extensions that were seen within the SMB2 traffic during the three-week capture period; following the \textit{smb2.filename} field. The easier to understand is Table~\ref{tab:commonSMB2FileExts}, which illustrates the number of common file extensions (e.g. doc, ppt, xls, pdf) that were part of the data. %The greatest point of note is that the highest percentage is ``.xml'' with $0.54$\%, which is found to be surprising result. -Originally we expected that these common file extensions would be a much larger total of traffic. However, as seen in Table~\ref{tab:commonSMB2FileExts}, these common file extensions were less than $2$\% of total files seen. The top ten extensions that we saw (Table~\ref{tab:top10SMB2FileExts}) comprised approximately $84$\% of the total seen. +Originally, we expected that these common file extensions would be a much larger total of traffic. However, as seen in Table~\ref{tab:commonSMB2FileExts}, these common file extensions were less than $2$\% of total files seen. The top ten extensions that we saw (Table~\ref{tab:top10SMB2FileExts}) comprised approximately $84$\% of the total seen. Furthermore, the majority of extensions are not readily identified. Upon closer examination of the tracing system it was determined that %these file extensions are an artifact of how Windows interprets file extensions. The Windows operating system merely guesses the file type based on the assumed extension (e.g. whatever characters follow after the final `.'). @@ -971,7 +974,7 @@ txt & 167,827 & 0.08 \\ \hline \subsection{Distribution Models} For simulations and analytic modeling, it is often useful to have models that describe storage systems I/O behavior. In this section, we attempt to map traditional probabilistic distributions to the data that we have observed. -Specifically, taking the developed CDF graphs, we perform curve fitting to determine the applicability of Gaussian and Weibull distributions to the the network filesystem I/O behavior. Note that an exponential distribution, typically used to model interarrival times and response times, is a special case of a Weibull distribution where $k=1$. +Specifically, taking the developed CDF graphs, we perform curve fitting to determine the applicability of Gaussian and Weibull distributions to the network filesystem I/O behavior. Note that an exponential distribution, typically used to model interarrival times and response times, is a special case of a Weibull distribution where $k=1$. Table~\ref{tbl:curveFitting} shows best-fit parametrized distributions for the measured data. % along with $R^2$ fitness values. %Based on the collected IAT and RT data, the following are the best fit curve representation equations with supporting $R^{2}$ values. In the case of each, it was found that the equation used to model the I/O behavior was a Gaussian equation with a single term. @@ -986,27 +989,54 @@ Table~\ref{tbl:curveFitting} shows best-fit parametrized distributions for the m % \item Read + Write command RT CDF, shown in Figure~\ref{fig:CDF-RT-RW}, has $R^2$ Value of $0.7837$. %\end{itemize} +%\begin{table*} +%\centering +%\begin{tabular}{|l|c|c|c||c|c|c|} +%\hline +%Model & \multicolumn{3}{|c|}{Gaussian} +% & \multicolumn{3}{|c|}{Weibull} \\ \hline +%CDF & \multicolumn{3}{|c|}{$\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\frac{x-\mu}{\sigma}}e^{\frac{-t^2}{2}}dt$} +% & \multicolumn{3}{|c|}{$1 - e^{(-x/\lambda)^k}$} \\ \hline \hline +%I/O Operation & $\mu$ & \multicolumn{2}{|c|}{$\sigma$} & $k$ & \multicolumn{2}{|c|}{$\lambda$} \\ \hline +%General RT & 3606.66$\pm$742.44 & \multicolumn{2}{|c|}{2.74931e+06$\pm$530} & 0.5652$\pm$0.0001 & \multicolumn{2}{|c|}{980.9721$\pm$0.4975} \\ +%General IAT & 786.72$\pm$2.79 & \multicolumn{2}{|c|}{10329.6$\pm$2} & 0.9031$\pm$0.0002 & \multicolumn{2}{|c|}{743.2075$\pm$0.2341} \\ +%Read RT & 44718.5$\pm$11715 & \multicolumn{2}{|c|}{1.72776e+07$\pm$8300} & 0.0004$\pm$0.0 & \multicolumn{2}{|c|}{1.5517$\pm$0.0028} \\ +%Read IAT & 24146$\pm$8062 & \multicolumn{2}{|c|}{1.189e+07$\pm$5700} & 0.0005$\pm$0.0 & \multicolumn{2}{|c|}{3.8134$\pm$0.0057} \\ +%Write RT & 379.823$\pm$2.809 & \multicolumn{2}{|c|}{4021.72$\pm$1.99} & 0.8569$\pm$0.0004 & \multicolumn{2}{|c|}{325.2856$\pm$0.2804} \\ +%Write IAT & 25785.7$\pm$8556.6 & \multicolumn{2}{|c|}{1.22491e+07$\pm$6000} & 0.0004$\pm$0.0 & \multicolumn{2}{|c|}{3.1287$\pm$0.0052} \\ +%Create RT & 502.084$\pm$5.756 & \multicolumn{2}{|c|}{21678.4$\pm$4.1} & 0.9840$\pm$0.0002 & \multicolumn{2}{|c|}{496.9497$\pm$0.1403} \\ +%Create IAT & 3694.82$\pm$1236.16 & \multicolumn{2}{|c|}{4.65553e+06$\pm$880} & 0.0008$\pm$0.0 & \multicolumn{2}{|c|}{2.3504$\pm$0.0009} \\ \hline +%%R+W RT & \textcolor{red}{0.8045} & \multicolumn{2}{|c|}{\textcolor{red}{0.2122}} & \textcolor{red}{5.103} & \multicolumn{2}{|c|}{\textcolor{red}{0.3937}} \\ \hline +%%R+W Byte Transfer & \textcolor{red}{0.3744} & \multicolumn{2}{|c|}{\textcolor{red}{0.2983}} & \textcolor{red}{1.153} & \multicolumn{2}{|c|}{\textcolor{red}{0.3937}} \\ +%Read Buff Transfer & 82.9179$\pm$0.7641 & \multicolumn{2}{|c|}{1117.9$\pm$0.54} & 1.0548$\pm$0.0003 & \multicolumn{2}{|c|}{85.2525$\pm$0.0575} \\ +%Write Buff Transfer & 46.2507$\pm$0.4475 & \multicolumn{2}{|c|}{640.621$\pm$0.316} & 1.0325$\pm$0.0004 & \multicolumn{2}{|c|}{46.8707$\pm$0.0328} \\ \hline +%\end{tabular} +%\caption{\label{tbl:curveFitting}Comparison of %$R^2$ +%$\mu$, $\sigma$, $k$, and $\lambda$ Values for Curve Fitting Equations on CDF Graphs} +%\vspace{-3em} +%\end{table*} + \begin{table*} \centering -\begin{tabular}{|l|c|c|c||c|c|c|} +\begin{tabular}{|l|r|c|c||r|c|c|} \hline Model & \multicolumn{3}{|c|}{Gaussian} & \multicolumn{3}{|c|}{Weibull} \\ \hline CDF & \multicolumn{3}{|c|}{$\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\frac{x-\mu}{\sigma}}e^{\frac{-t^2}{2}}dt$} & \multicolumn{3}{|c|}{$1 - e^{(-x/\lambda)^k}$} \\ \hline \hline I/O Operation & $\mu$ & \multicolumn{2}{|c|}{$\sigma$} & $k$ & \multicolumn{2}{|c|}{$\lambda$} \\ \hline -General RT & 3606.66$\pm$742.44 & \multicolumn{2}{|c|}{2.74931e+06$\pm$530} & 0.5652$\pm$0.0001 & \multicolumn{2}{|c|}{980.9721$\pm$0.4975} \\ -General IAT & 786.72$\pm$2.79 & \multicolumn{2}{|c|}{10329.6$\pm$2} & 0.9031$\pm$0.0002 & \multicolumn{2}{|c|}{743.2075$\pm$0.2341} \\ -Read RT & 44718.5$\pm$11715 & \multicolumn{2}{|c|}{1.72776e+07$\pm$8300} & 0.0004$\pm$0.0 & \multicolumn{2}{|c|}{1.5517$\pm$0.0028} \\ -Read IAT & 24146$\pm$8062 & \multicolumn{2}{|c|}{1.189e+07$\pm$5700} & 0.0005$\pm$0.0 & \multicolumn{2}{|c|}{3.8134$\pm$0.0057} \\ -Write RT & 379.823$\pm$2.809 & \multicolumn{2}{|c|}{4021.72$\pm$1.99} & 0.8569$\pm$0.0004 & \multicolumn{2}{|c|}{325.2856$\pm$0.2804} \\ -Write IAT & 25785.7$\pm$8556.6 & \multicolumn{2}{|c|}{1.22491e+07$\pm$6000} & 0.0004$\pm$0.0 & \multicolumn{2}{|c|}{3.1287$\pm$0.0052} \\ -Create RT & 502.084$\pm$5.756 & \multicolumn{2}{|c|}{21678.4$\pm$4.1} & 0.9840$\pm$0.0002 & \multicolumn{2}{|c|}{496.9497$\pm$0.1403} \\ -Create IAT & 3694.82$\pm$1236.16 & \multicolumn{2}{|c|}{4.65553e+06$\pm$880} & 0.0008$\pm$0.0 & \multicolumn{2}{|c|}{2.3504$\pm$0.0009} \\ \hline +General RT & 3606.66$\pm$20.6\% & \multicolumn{2}{|r|}{2.74931e+06$\pm$0.02\%} & 0.5652$\pm$0.02\% & \multicolumn{2}{|r|}{980.9721$\pm$0.05\%} \\ +General IAT & 786.72$\pm$0.35\% & \multicolumn{2}{|r|}{10329.6$\pm$0.02\%} & 0.9031$\pm$0.02\% & \multicolumn{2}{|r|}{743.2075$\pm$0.02\%} \\ +Read RT & 44718.5$\pm$26.2\% & \multicolumn{2}{|r|}{1.72776e+07$\pm$0.05\%} & 0.0004$\pm$0.00\% & \multicolumn{2}{|r|}{1.5517$\pm$0.18\%} \\ +Read IAT & 24146$\pm$33.4\% & \multicolumn{2}{|r|}{1.189e+07$\pm$0.05\%} & 0.0005$\pm$0.00\% & \multicolumn{2}{|r|}{3.8134$\pm$0.15\%} \\ +Write RT & 379.823$\pm$0.74\% & \multicolumn{2}{|r|}{4021.72$\pm$0.05\%} & 0.8569$\pm$0.05\% & \multicolumn{2}{|r|}{325.2856$\pm$0.09\%} \\ +Write IAT & 25785.7$\pm$33.2\% & \multicolumn{2}{|r|}{1.22491e+07$\pm$0.05\%} & 0.0004$\pm$0.00\% & \multicolumn{2}{|r|}{3.1287$\pm$0.17\%} \\ +Create RT & 502.084$\pm$1.15\% & \multicolumn{2}{|r|}{21678.4$\pm$0.02\%} & 0.9840$\pm$0.02\% & \multicolumn{2}{|r|}{496.9497$\pm$0.03\%} \\ +Create IAT & 3694.82$\pm$33.5\% & \multicolumn{2}{|r|}{4.65553e+06$\pm$0.02\%} & 0.0008$\pm$0.00\% & \multicolumn{2}{|r|}{2.3504$\pm$0.04\%} \\ \hline %R+W RT & \textcolor{red}{0.8045} & \multicolumn{2}{|c|}{\textcolor{red}{0.2122}} & \textcolor{red}{5.103} & \multicolumn{2}{|c|}{\textcolor{red}{0.3937}} \\ \hline %R+W Byte Transfer & \textcolor{red}{0.3744} & \multicolumn{2}{|c|}{\textcolor{red}{0.2983}} & \textcolor{red}{1.153} & \multicolumn{2}{|c|}{\textcolor{red}{0.3937}} \\ -Read Buff Transfer & 82.9179$\pm$0.7641 & \multicolumn{2}{|c|}{1117.9$\pm$0.54} & 1.0548$\pm$0.0003 & \multicolumn{2}{|c|}{85.2525$\pm$0.0575} \\ -Write Buff Transfer & 46.2507$\pm$0.4475 & \multicolumn{2}{|c|}{640.621$\pm$0.316} & 1.0325$\pm$0.0004 & \multicolumn{2}{|c|}{46.8707$\pm$0.0328} \\ \hline +Read Buff Transfer & 82.9179$\pm$0.92\% & \multicolumn{2}{|r|}{1117.9$\pm$0.05\%} & 1.0548$\pm$0.03\% & \multicolumn{2}{|r|}{85.2525$\pm$0.07\%} \\ +Write Buff Transfer & 46.2507$\pm$0.97\% & \multicolumn{2}{|r|}{640.621$\pm$0.05\%} & 1.0325$\pm$0.04\% & \multicolumn{2}{|r|}{46.8707$\pm$0.07\%} \\ \hline \end{tabular} \caption{\label{tbl:curveFitting}Comparison of %$R^2$ $\mu$, $\sigma$, $k$, and $\lambda$ Values for Curve Fitting Equations on CDF Graphs} @@ -1036,11 +1066,13 @@ $\mu$, $\sigma$, $k$, and $\lambda$ Values for Curve Fitting Equations on CDF Gr %Examination of the Response Time (RT) and Inter Arrival Times (IAT) revealed the speed and frequency with which metadata operations are performed, as well as the infrequency of individual users and sessions to interact with a given share. %% NEED: Run the matlab curve fitting to complete this section of the writing -Our comparison of the existing standard use of a exponential distribution to model network inter-arrival and response times is still valid. One should notice that the Gaussian distributions +%Our comparison of the existing standard use of an exponential distribution to model network inter-arrival and response times is still valid. +As indicated by the error bounds, one can see that the Weibull distributions are generally much better fits for all the different distributions. Furthermore, the Poisson arrival assumption (exponential distribution $k=1$) is only valid for general I/O, but the write and create service times are exponential. Interestingly, the read and write buffer sizes are very close to exponential, though a Gaussian distribution is a close fit. +%One should notice that the Gaussian distributions % had better $R^2$ result than the exponential equivalent for write operations. This is not surprising due to the step-function shape of the Figure~\ref{fig:CDF-RT-Write} CDF. Examining the $R^2$ results for the read + write I/O operations we find that the exponential distribution is far more accurate at modeling this combined behavior. -for write and create operations are similar, while those for read operations are not. Further more there is less similarity between the modeled behavior of general operation inter arrival times and their response times, showing the need for a more refined model for each aspect of the network filesystem interactions. -One should also notice that the general operation model is more closely similar to that of the creates. -This makes sense since create operations are found to dominate the network filesystem I/O, which aligns well with the number of existing close operations. +The models for the write and create operations are similar, while those for read operations are not. Furthermore, there is less similarity between the modeled behavior of general operation inter arrival times and their response times, showing the need for a more refined model for each aspect of the network filesystem interactions. +%One should also notice that the general operation model is more closely similar to that of the creates. +%This makes sense since create operations are found to dominate the network filesystem I/O, which aligns well with the number of existing close operations. %improves the ability of a exponential distribution to model the combined behavior.} %Observations: %\begin{itemize} @@ -1062,13 +1094,13 @@ This makes sense since create operations are found to dominate the network files % \item Writes cause the largest amount of data to be passed over the network. While Read operations occur at the largest number and cause the larger total number of bytes to be transferred, write operations are more expensive by an order of magnitude. % \item \textcolor{red}{Here will be observation on the modeling of poisson fit.} %\end{itemize} -Due to the large number of metadata operations, the use of smart storage solutions could be used to minimize the impact of these I/O. Smart storage elements can aid by performing metadata operations without the need to access persistent storage, thus causing shorter response times. In this manner, the use of smart storage can also help reduce bottlenecks with larger network filesystems and minimize the effect of traffic on overall network performance. + \subsection{System Limitations and Challenges} \label{System Limitations and Challenges} When initially designing the tracing system used in this paper, different aspects were taken into account, such as space limitations of the tracing system, packet capture limitations (e.g. file size), and speed limitations of the hardware. One limitation encountered in the packet capture system deals with the functional pcap (packet capture file) size. The concern being that the pcap files only need to be held until they have been filtered for specific protocol information and then compressed using the DataSeries format, but still allow for room for the DataSeries files being created to be stored. %Other limitation concerns came from the software and packages used to collect the network traffic data~\cite{Orosz2013,dabir2007bottleneck,skopko2012loss}. These ranged from timestamp resolution provided by the tracing system's kernel~\cite{Orosz2013} to how the packet capturing drivers and programs (such as dumpcap and tshark) operate along with how many copies are performed and how often. The speed limitations of the hardware are dictated by the hardware being used (e.g. Gb capture interface) and the software that makes use of this hardware (e.g. PF\_RING). %After all, our data can only be as accurate as the information being captured~\cite{seltzer2003nfs,anderson2004buttress}. -An other concern was whether or not the system would be able to function optimally during periods of high network traffic. All aspects of the system, from the hardware to the software, have been altered to help combat these concerns and allow for the most accurate packet capturing possible. +Another concern was whether or not the system would be able to function optimally during periods of high network traffic. All aspects of the system, from the hardware to the software, have been altered to help combat these concerns and allow for the most accurate packet capturing possible. %About Challenges of system %While the limitations of the system were concerns, there were other challenges that were tackled in the development of this research. @@ -1082,7 +1114,7 @@ Because the data is being collected from an active network, there will be differ %Normally, one could simply re-perform the conversion process to a DataSeries file, but due to the rate of the packets being captured and security concerns of the data being captured, we are unable to re-run any captured information. \section{Conclusions and Future Work} -Our analysis of this university network filesystem illustrated the current implementation and use of the CIFS/SMB protocol in a large academic setting. We notice the effect of caches on the ability of the filesystem to limit the number of accesses to persistant storage. The effect of enterprise storage disks access time can be seen in the response time for read and write I/O. Metadata operations dominate the majority of network communication, which is of less surprise since SMB is a known chatty protocol. We do notice that the CIFS/SMB protocol continues to be chatty with metadata I/O operations regardless of the version of SMB being implemented; $74.66$\% of I/O being metadata operations for SMB2. +Our analysis of this university network filesystem illustrated the current implementation and use of the CIFS/SMB protocol in a large academic setting. We notice the effect of caches on the ability of the filesystem to limit the number of accesses to persistent storage. The effect of enterprise storage disks access time can be seen in the response time for read and write I/O. Metadata operations dominate the majority of network communication, which is of less surprise since SMB is a known chatty protocol. We do notice that the CIFS/SMB protocol continues to be chatty with metadata I/O operations regardless of the version of SMB being implemented; $74.66$\% of I/O being metadata operations for SMB2. We also find that read and write transfer sizes are significantly smaller than would be expected and requires further study as to the impact on current storage systems. %operations happen in greater number than write operations (at a ratio of 1.06) and the size of their transfers are is also greater by a factor of about 2. %However, the average write operation includes a larger number of relatively smaller writes. @@ -1094,8 +1126,8 @@ However, the general I/O can be modeled using the same standard; which has simil %\subsection{Future Work} The analysis work will eventually incorporate oplocks and other aspects of resource sharing on the network to gain a more complete picture of the network's usage and bottlenecks. Network filesystem usage from an individual user scope has become simple and does not contain a greater deal of read, write, and create operations. -Further analysis will be made in examining how the determined metrics change when examined at the scope of a per share (i.e. TID) or per user (i.e. UID). At this level of examination we will be able to obtain a better idea of how each share is interacted with, as well as how files and directories are shared and access control is implemented. - +Further analysis will be made in examining how the determined metrics change when examined at the scope of a per share (i.e. TID) or per user (i.e. UID). At this level of examination, we will be able to obtain a better idea of how each share is interacted with, as well as how files and directories are shared, and access control is implemented. +Due to the large number of metadata operations, the use of smart storage solutions could be used to minimize the impact of these I/O. Smart storage elements can aid by performing metadata operations without the need to access persistent storage, thus causing shorter response times. In this manner, the use of smart storage can also help reduce bottlenecks with larger network filesystems and minimize the effect of traffic on overall network performance. %\end{document} % This is where a 'short' article might terminate %ACKNOWLEDGMENTS are optional